Simple Data Engineering in Python 3.5+ — Pycon.DE 2017 Karlsruhe — Bonobo ETL
•
1 j'aime•618 vues
Simple Data Engineering in Python 3.5+ using Bonobo ETL, with real world example using Django2 and DBPedia. https://www.bonobo-project.org/ Presentation from Pycon.DE 2017 in Karlsruhe
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Simple Data Engineering in Python 3.5+ — Pycon.DE 2017 Karlsruhe — Bonobo ETL
Similaire à Simple Data Engineering in Python 3.5+ — Pycon.DE 2017 Karlsruhe — Bonobo ETL (20)
Dernier
Dernier (20)
Simple Data Engineering in Python 3.5+ — Pycon.DE 2017 Karlsruhe — Bonobo ETL
- 1. Bonobo ETL Data Engineering for Humans Python 3.5+
- 2. quick intro 101 one minute to jump in 102 theory and concepts 103 apply to reality wrap up
- 3. Intro
- 4. Extract Transform Load foo bar baz Extract Transform Load
- 5. Extract Transform Load foo bar baz Extract Transform Load Transform more Join DB HTTP POST log?
- 7. Why don’t we have… • Extract Transform Load using code as configuration. • Preferably Python code. • Something that can be tested (I mean, by a machine). • Something that can use inheritance. • Quick install on a laptop, great on servers too.
- 8. Meet Bonobo
- 9. Bonobo is… A framework to write ETL jobs …using code as configuration …with the same concepts as legacy ETLs. It’s just Python!
- 10. Bonobo is not… …a data analysis or statistical toolkit …a scheduler or dependency manager …a big data tool
- 11. </tl;dr>
- 12. Action
- 13. 101 jump in
- 14. $ pip install bonobo
- 15. $ bonobo init pycon
- 16. $ bonobo run pycon
- 17. - extract in=1 out=42 - transform in=42 out=21 - load in=21
- 18. 102 theory
- 19. Graphs…
- 20. Graphs… import bonobo graph = bonobo.Graph() graph.add_chain( extract, transform, load, )
- 21. Transformations…
- 22. Functions items = {...} def get_item(id): return id, items.get(id) 1 In / 1 Out
- 23. Generators orders = {...} def get_orders(user): yield from orders.get(user) 1 In / 0-n Out
- 24. Iterators numbers = range(2017) messages = [ ('Bonjour', 'Paris', ), ('Ciao', 'Rimini', ), ('Guten Tag', 'Karlsruhe', ), ] 0 In / 0-n Out
- 25. Classes class ExtractMessages: def __call__(self): yield 'Bonjour', 'Paris', {'year': 2017} yield 'Ciao', 'Rimini', {'year': 2017} yield 'Guten Tag', 'Karlsruhe', {'year': 2017}
- 26. … anything, as long as it’s callable().
- 27. 103 reality
- 28. 1 • Bonobo meets Django2 2 • Hello, DBPedia 3 • Music groups… 4 • Music genres… 5 • Links and play!
- 29. 1 • Bonobo meets Django2 2 • Hello, DBPedia 3 • Music groups… 4 • Music genres… 5 • Links and play!
- 30. DBPedia & SPARQL Triplets : (SUBJECT, PREDICATE, OBJECT) DBPedia : Wikipedia as triplets SPARQL : Query language for triplet stores. ( )
- 31. 1 • Bonobo meets Django2 2 • Hello, DBPedia 3 • Music groups… 4 • Music genres… 5 • Links and play!
- 32. 1 • Bonobo meets Django2 2 • Hello, DBPedia 3 • Music groups… 4 • Music genres… 5 • Links and play!
- 33. 1 • Bonobo meets Django2 2 • Hello, DBPedia 3 • Music groups… 4 • Music genres… 5 • Links and play!
- 34. Wrap up
- 35. State of Bonobo ETL • First commit : December 2016 • 25 releases, ~565 commits, 12 contributors • Current « stable » 0.5.1 • Target : 1.0 early 2018 when it’s ready
- 36. Small scale … < Tb One minute to install. Easy to deploy It is a Lean Manufacturing Toolkit for Data.
- 37. www.bonobo-project.org Data Engineering for Humans @bonobo_etl
- 41. Oh, wait!
- 42. Sprint!
- 43. Stickers!
- 44. Feedback!