OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan
•
0 j'aime•166 vues
Integrating microservices and taming distributed systems is hard. In this talk I will present three challenges we observed in real-life projects and discuss how to avoid them using Open Source orchestration. Communication is complex. With everything being distributed failures are normal, so you need sophisticated failure handling strategies (e.g. stateful retry). A synchronicity requires you to handle timeouts. This is not only about milliseconds, systems get much more resilient when you can wait for minutes, hours or even longer. Distributed transactions cannot simply be delegated to protocols like XA. So, you need to solve the requirement to retain consistency in case of failures. I will not only use slides but demonstrate concrete source code examples available on GitHub.
Recommandé
Recommandé
Contenu connexe
Similaire à OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan
Similaire à OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan (20)
Dernier
Dernier (20)
OSDC 2019 | The Benefits of Orchestration in Distributed Systems by Niall Deehan
- 1. The Benefits of Orchestration in Distributed Systems @NiallDeehan
- 2. Who I am? Niall Deehan
- 3. Distributed systems Challenges of asynchronicity Distributed Transactions Communication is complex
- 4. Do A Do B All or nothing + try { tx.begin(); doA(); doB(); tx.commit(); } catch (Exception e) { tx.rollback(); } @Transactional public void createCustomer(Customer cust) { // ... } Or simply: Once upon a time:
- 8. Some Service Some Service Some Service Some Service Some Service Some Service Some Service Failure will happen. Accept it! But keep it local! Be resilient.
- 10. Let‘s start with a simple example Credit Card Payment REST
- 12. It‘s a blocking request. It doesn‘t scale
- 13. Starbucks does not use two phase commit Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html Photo by John Ingle
- 14. That means Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Local ACID Local ACID 1 (micro-)service 1 aggregate 1 program 1 resource Violates „I“ of ACID
- 15. Youmightknowthisfrom: Do A Do B Temporarily inconsistent Eventually consistent again t Consistent Photo by Gerhard51, available under Creative Commons CC0 1.0 license.
- 16. It is impossible to differentiate certain failure scenarios: Independant of communication style! Service Provider Client
- 18. Let‘s start with a simple example Credit Card Payment REST
- 19. Circuit Breaker Photo by CITYEDV, available under Creative Commons CC0 1.0 license.
- 22. Fail fast is important but not enough!
- 23. Berlin, Germany bernd.ruecker@camunda.com @berndruecker Bernd Ruecker Co-founder and Developer Advocate of Camunda
- 24. Photo by Tookapic, available under Creative Commons CC0 1.0 license.
- 25. Translation : „There was an error while sending your boarding pass“
- 31. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Bernd
- 32. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Bernd
- 33. Check-in Barcode Generator Web-UI Output Mgmt Current situation – the bad part Stateful Retry Bernd
- 35. We are having some technical difficulties and cannot present you your boarding pass right away. But we do actively retry ourselves, so lean back, relax and we will send it on time.
- 36. Check-in Barcode Generator Web-UI Output Mgmt Possible situation – much better! Bernd
- 37. Check-in Barcode Generator Web-UI Output Mgmt Possible situation – much better! Stateful Retry Bernd
- 38. Check-in Barcode Generator Web-UI Output Mgmt Stateful Retry Possible situation – much better! The failure never leaves this scope! Bernd
- 39. Persist thing (Entity, Document, Actor, …) State machine or workflow engine Typical concerns DIY = effort, accidental complexity Complex, proprietary, heavyweight, slow, don‘t scale, developer adverse Scheduling, Versioning, operating, visibility, scalability, … Handling State
- 41. Workflow engines, state machines It is relevant in modern architectures
- 42. What about open source…?
- 43. CADENCE Silicon valley has recognized Workflow engines, state machines
- 44. What about purpose built and open source…
- 46. CADENCE also at scale Workflow engines, state machines
- 47. CADENCE for todays demo Workflow engines, state machines
- 50. Payment Now you have a state machine! Credit Card REST
- 51. has to implement Retry has to implement Idempotency Client Service Provider
- 52. We are processing your payment. Do not leave this page. And for god sake – do not reload! It is a business problem anyway!
- 53. We are processing your payment. Do not leave this page. And for god sake – do not reload! It is a business problem anyway! We are currently processing your request. Don‘t worry, it will happen safely – even if you loose connection. Feel free to reload this page any time!
- 54. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
- 56. It is impossible to differentiate certain failure scenarios. Independant of communication style! Service Provider Client
- 57. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
- 58. Distributed systems introduce complexity you have to tackle! Credit Card Payment REST
- 60. „The customer wants a synchronous response“ Check-in Barcode Generator Web-UI Bernd Output Mgmt „Eh – no!“
- 61. generateBoardingPass HTTP 200 OK HTTP 202 ACCEPTED Check-in A synchronous response is possible in the happy case, otherwise it is switched to asynchronous processing.
- 64. Check-in Barcode Generator Web-UI Bernd Output Mgmt Asynchronous communication You need to monitor timeouts
- 65. Check-in Barcode Generator Web-UI Bernd Output Mgmt Remember… The failure never leaves this scope!
- 66. Workflow…
- 67. Workflow…
- 68. BPMN Business Process Model and Notation ISO Standard
- 69. BPMN Executable and mature Easy to understand* Widespread *(for Biz and Dev)
- 71. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
- 72. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
- 73. Clemens Vasters Architect at Microsoft http://vasters.com/archive/Sagas.html
- 74. Living documentation for long-running behaviour
- 75. Visual HTML reports for test cases
- 77. Compare to e.g. Step Functions
- 78. Imagine more complex stuff
- 79. has to implement Timeout, Retry has to implement Idempotency Client Service Provider
- 82. Avoids thundering herds Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license.
- 83. Photo by Gopal Vijayaraghavan, available under Creative Commons BY 2.0 license. https://aws.amazon.com/de/message/5467D2/
- 86. Compensation – the classical example Saga book hotel book car book flight cancel hotel cancel car 1. 2. 3. 5.6. In case of failure trigger compensations book trip
- 87. 2 alterntive approaches: choreography & orchestration
- 89. Event-driven choreography Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled Perform undo (cancel car booking) Perform undo (cancel hotel)
- 90. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
- 91. Implementing changes in the process Hotel Flight Car Trip Trip failed Trip requested Hotel booked Car booked Request trip Flight failed Car canceled Hotel canceled You have to adjust all services and redeploy at the same time! We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first!
- 92. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License
- 94. Orchestration Hotel Flight Car Trip Trip booked Request trip Book hotel Hotel booked Car booked Flight booked Book car Book flight We have a new basic agreement with the car rental agency and can cancel for free within 1 hour – do that first! You have to adjust one service and redeploy only this one!
- 95. Describe orchestration with BPMN Trip Trip booked Request trip
- 96. The workflow is part of the service Trip
- 97. The workflow is part of the service Trip Payment
- 99. has to implement Timeout, Retry, Compensation has to offer Compensation has to implement Idempotency Client Service Provider
- 100. has to implement Timeout, Retry, Compensation has to offer Compensation has to implement Idempotency Client Service ProviderDon‘t forget about state
- 101. # Be aware of complexity of distributed systems # Know strategies and tools to handle it e.g. Circuit breaker (Hystrix) Workflow engine for stateful retry, waiting, timeout and compensation (Camunda)
- 102. Thank you!