Introduction to Data Engineering
•
5 j'aime•2,829 vues
This presentation explains what data engineering is and describes the data lifecycles phases briefly. I used this presentation during my work as an on-demand instructor at Nooreed.com
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Introduction to Data Engineering
Similaire à Introduction to Data Engineering (20)
Plus de Hadi Fadlallah
Plus de Hadi Fadlallah (20)
Dernier
Dernier (20)
Introduction to Data Engineering
- 1. Introduction to Data Engineering Hadi Fadlallah
- 2. About Me … … … … Just a Data Engineer! 4/6/2022 2
- 3. Plan • What is Data Engineering? • Data Engineer vs. Data Scientist vs. Data Analyst • Understanding Data Management (Data Layers, DQS, MDS, Provenance) • Distributed Computing • Designing Data Pipelines (Choosing Paradigm / Technologies) • Data Engineer Jobs / Required Skills • Helpful Tips • Online Courses 4/6/2022 3
- 5. What is Data Engineering? 4/6/2022 5 • Data Engineering is the act of: • Collecting data • Transforming (…) data • Validating data “Making data consumable”
- 6. What is a Data Engineer? AI/ML Engineer BI Developer Data Analyst Database Administrator Report Developer Data Developer Data Architect Data Integration Specialist ETL Developer Data Scientist 4/6/2022 6
- 7. Data Engineer vs. Data Scientist 4/6/2022 7 Source: https://elu.nl/careers-in-data-science-data-analyst-vs-data-engineer-vs-data-scientist/
- 8. 4/6/2022 8
- 9. 4/6/2022 9 Data Sources Processed Data Storage Data Ingestion Raw Data Storage Data Processing Data Visualization Understanding Data Layers
- 11. 4/6/2022 11 Source: https://www.dataentryoutsourced.com/blog/ever-changing-master-data-management/
- 12. Data Provenance 4/6/2022 12 Source: https://www.canto.com/blog/data-lineage/
- 13. Data Wrangling vs. Data Pre-processing 4/6/2022 13 Source: https://medium.com/swlh/data-pre-processing-data-wrangling-4a6a8624e747 Data Pre-processing Data Wrangling
- 14. What are Distributed Systems? 4/6/2022 14
- 15. 15/20 • 1 Cluster = many machines (nodes) • Parallel computing • Scalability • Fault-tolerance (High availability)
- 16. 16/20 Data ingestion Data storage Data Processing
- 18. Understanding Data & Requirements 4/6/2022 18 Source: https://firebrand.training/uk/lea rn/pmp/course-material/project- scope-management/collect- requirements
- 19. Choosing a Paradigm 4/6/2022 19
- 24. Relational databases 4/6/2022 24 Source: https://www.stambia.com/en/solutions/by-technology/integration-to-database/relational-database-rdbms
- 25. Data Integration 4/6/2022 25 Source: https://deevita.com/top-data-integration-tools/
- 26. NoSQL databases 4/6/2022 26 Source: https://dock.ie/data.html
- 27. 4/6/2022 27 Big Data ecosystem Source: https://sranka.wordpress.com/2014/01/29/big-data-technologies/
- 28. 4/6/2022 28 Data Engineer Jobs
- 29. 4/6/2022 29 Stack Overflow: https://stackoverflow.com/jobs?q=data+engineer LinkedIn: https://www.linkedin.com/jobs/search/?geoId=92000001&keywords=data%20engineer&location=Remote
- 30. Required Skills 2. SQL, Python 3. Pandas library 4. AWS, Azure 5. Relational databases (SQL Server and MySQL) 6. Hadoop, Spark, Hive 7. NOSQL databases (MongoDB, Neo4j, and Cassandra) … 4/6/2022 30
- 31. 4/6/2022 31 1st Required Skill
- 33. 4/6/2022 33
- 34. 4/6/2022 34 First, make it run correctly (in an acceptable way). Then start improving performance
- 35. 4/6/2022 35 Again, make sure that you understand well the data and the requirements!
- 36. 4/6/2022 36
- 37. DON’T use Big Data technologies just because you have a huge volume of data. Follow best practices 4/6/2022 37
- 39. 4/6/2022 39 • Coursera: • Google Cloud - Data Engineering, Big Data, and Machine Learning on GCP Specialization • San Diego - Big Data Specialization • Udacity: • Data Engineering nanodegree • DataCamp: • Data Engineer with Python Track • IBM – CognitiveClass.ai • Free data science and data engineering courses • Udemy: • Data Science A-Z™: Real-Life Data Science Exercises Included