Optimize your data pipeline without rewriting it - Bigdata tech warsaw 2020
•
1 j'aime•61 vues
This document discusses optimizing a data pipeline without rewriting code. It recommends visualizing the process, breaking it down into parallel tasks, removing idle time between tasks, identifying bottlenecks, and focusing optimization efforts there. The overall process is to understand requirements, break work into parallel tasks, remove inefficiencies, identify bottlenecks, optimize those areas, and stop when performance is good enough.
Recommandé
Recommandé
Contenu connexe
Similaire à Optimize your data pipeline without rewriting it - Bigdata tech warsaw 2020
Similaire à Optimize your data pipeline without rewriting it - Bigdata tech warsaw 2020 (20)
Dernier
Dernier (20)
Optimize your data pipeline without rewriting it - Bigdata tech warsaw 2020
- 1. Optimize your data pipeline without rewriting it Magnus Runesson
- 2. “How did you do that?”
- 3. Who am I?
- 4. We are the rails and brains of open banking Trusted by the industry leaders • Natwest • BNP Paribas Fortis • ABN AMRO • PayPal • Klarna • SEB 260 Employees across Europe Local offices in • Sweden • Finland • Denmark • United Kingdom • Germany • Netherlands • Spain • France • Poland 2,500 + banks & FIs connected We’re connected for access to all types of accounts, from banks, neo-banks, credit cards and more, and bring them together for you via a single, beautiful API. Industry Authority Member of EU, Berlin Group & Open Banking advisory boards ISO/IEC 27001 Certified PSD2 Licensed 2,000 + Platform users
- 5. Agenda Introduction The problem The process to success Summary Proprietary & confidential
- 6. It is not fast enough!
- 7. The process
- 8. Visualize
- 9. Database DB to files on storage bucket Load to bigquery Database Google BigQuery
- 10. 00 03 06 09 12 15 18 21 DB to files on storage bucket Load files to bigquery
- 13. Requirements
- 14. Break down
- 15. 00 03 06 09 12 15 18 21 DB to files on storage bucket Load files to bigquery
- 16. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 17. Parallelise
- 18. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 19. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 20. Remove idle time
- 21. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 22. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 23. Find bottleneck
- 24. Table 1 Table 2 Table 3 Table 4 Table 1 Table 2 Table 3 Tab 4 00 03 06 09 12 15 18 21
- 25. Task 1 Task 2 Task 6Task 3 Task 5 Task 4 Time Bottleneck example
- 26. Task 1 Task 2 Task 6Task 3 Task 5 Task 4 Time Bottleneck example - critical path
- 27. Zoom in
- 28. Process recap
- 29. Visualize Understand requirements Break down Parallelise Remove idle time Find bottlenecks Zoom in
- 30. Remember ● Use dependency based scheduling ● Look on the metrics ● Use metrics to make your stand ● Work with the bottleneck once entering the code ● Stop when good enough
- 31. Thank you! (Psst… come work with us!) Magnus.Runesson@tink.se @MRunesson