Let's Build a Service Oriented Data Pipeline!

•Télécharger en tant que PPTX, PDF•

0 j'aime•2,546 vues

Like many software projects, data pipelines built by Business Intelligence teams often start out as quickly built monoliths, but over time they can be made simpler and more maintainable by splitting them into a series of loosely coupled jobs, stealing ideas from service oriented architecture. In this presentation we'll look at the challenges we've faced scaling Data Analytics at Hootsuite, then move into a live coding session, where we'll stitch together a data pipeline as a series of Scala apps, deployed to AWS Lambda, connected using Airbnb's open source Airflow tool.

Technologie

Let’s Build a Service Oriented
Data Pipeline!
June 2016
Software Developer | Hootsuite
Yasha Podeswa

Now: Software Developer at Hootsuite
Me!

Introduce a problem that requires a new data pipeline
Design it in a service oriented style
Build it on stage!
This Talk

Passive Aggressive Inc. just
cancelled their subscription!
Desperate Dan in trouble!
The Problem

What We’re Starting With
Things Users
Did

What We’re Starting With
Things
Organizations
Did

High Level Plan
JSON files
Calculate stats
about
organizations
DB

High Level Plan
JSON files
Calculate stats
about
organizations
DB
Extract
Transform
Load

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Useful for lots of
things!

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Shouldn’t run until
dependent job done

Need a “Service” Communication and
Orchestration Layer!

First App Event Cleaning and Loading
Read logs from S3, clean and sort into different types of
events, load into data warehouse
Vanilla Scala app
AWS Lambda

Second App Organization Stat Calculation
Read cleaned/sorted events from data warehouse, calculate
stats about organization, load stats to data warehouse
Vanilla Scala app
AWS Lambda

Third App Airflow
Hook up the Lambda apps in a dependency graph
● Scheduling
● Retries
● Monitoring

Steal my code!
https://github.com/yashap/etl-load-events
https://github.com/yashap/etl-organization-stats
https://github.com/yashap/airflow

Contenu connexe

En vedette

Integrating NiFi and Flink

Bryan Bende

Data is being collected more and more every year. Cloud applications, including IoT, web, and mobile send torrents of bits at our data centers that have to be processed and stored. In addition, users expect an always-on experience, with little room for error. Numerous companies are successfully doing this every day. In this webinar, you will learn about the convergence of complementary technologies: Spark, Mesos, Akka, Cassandra and Kafka (SMACK), how Apache Kafka can help you get your data under control and the critical role Kafka plays in your data pipeline. Webinar recording: https://youtu.be/uwYlwLyv-1s Webinar Q&A will be posted shortly.

Webinar - How to Build Data Pipelines for Real-Time Applications with SMACK &...

DataStax

Writing a Jenkins / Hudson plugin

Anthony Dahanne

Curator intro

Jordan Zimmerman

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

Building an Event-oriented Data Platform with Kafka, Eric Sammer

confluent

In this session, we will cover our experience working with Apache NiFi, an easy to use, powerful, and reliable system to process and distribute a large volume of data. The first part of the session will be an introduction to Apache NiFi. We will go over NiFi main components and building blocks and functionality. In the second part of the session, we will show our use case for Apache NiFi and how it's being used inside our Data Processing infrastructure.

Data ingestion and distribution with apache NiFi

Lev Brailovskiy

This is the slide deck from my keynote at the EA User Event in Brussels, September 2015. Micro-services and micro-services architecture are the next hype in software development. Websites and blogs are full of introducing posts, the first books are being written and the first conferences organized. There’s big promises of scalability, flexibility and replaceability of individual elements in your landscape. However, when you are knee deep in the mud as a software architect at an insurance, it is very hard to find help on how to design applications and components in a micro-services architecture. During this talk Sander will show how he used Enterprise Architect to model the micro services architecture, and will explain the difficulties and the lessons learned, using many real-life examples.

Modeling Microservices

Sander Hoogendoorn

Building a real-time streaming platform using Kafka Connect + Kafka Streams

confluent

Apache kafka

Rahul Jain

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Introduction to Kafka Streams

Guozhang Wang

Introduction to Apache Kafka

Jeff Holoman

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rim...

confluent

reveal.js 3.0.0

Hakim El Hattab

Introduction to Kafka and Zookeeper

Rahul Jain

En vedette (14)

Integrating NiFi and Flink

Webinar - How to Build Data Pipelines for Real-Time Applications with SMACK &...

Writing a Jenkins / Hudson plugin

Curator intro

Building an Event-oriented Data Platform with Kafka, Eric Sammer

Data ingestion and distribution with apache NiFi

Modeling Microservices

Building a real-time streaming platform using Kafka Connect + Kafka Streams

Apache kafka

Introduction to Kafka Streams

Introduction to Apache Kafka

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rim...

reveal.js 3.0.0

Introduction to Kafka and Zookeeper

Similaire à Let's Build a Service Oriented Data Pipeline!

SPS Warsaw 2017 - Building applications with PowerApps, Microsoft flow and Of...

Ahmad Najjar

Microsoft Flow is a brand new SaaS offering, for automating workflows across the growing number of applications and SaaS services that business users rely on. This session introduces Microsoft Flow and how it can help you save time, and make your business highly visible by partnering with Microsoft's growing ecosystem including O365 and SharePoint Online. It demonstrates how you can create an automated workflow between your favorite O365 apps and services to get notifications, synchronize files, collect data, and more!

SPS Brno 2017 - Go with the Microsoft flow

Ahmad Najjar

Introduction to Microsoft Power BI

CCG

Jira - Solving Reporting Problems using eazyBI

Marlon Palha

Synapse NanoApps

Synapse Information Ltd

Microsoft Business Intelligence - Practical Approach & Overview

Li Ken Chong

Intro to PowerApps and Flow

April Dunnam

Microsoft Flow is a new service (currently in preview) that helps you to build connection between different services and apps. You can create several workflows using available templates in O365 to do specific tasks like sending e-mail notifications to an app or sending messages at specific schedules. In this session you will get an overview of what Microsoft Flow can do for you and how to create your own custom workflows easily using easy to learn visual designer tools.

Automating your tasks with microsoft flow

Dipti Chhatrapati

Power BI has in its DNA the goal of enabling everybody to experience their data any way, anywhere—in seconds and at global scale. Power BI offers a set of capabilities that are uniquely enabled by its global and cloud nature: The ability to harness data from Excel spreadsheets, on-premises data sources through the data gateway, big data, streaming data, and cloud services. It doesn’t matter what type of data you want or where it lives, Power BI allows you to connect to hundreds of data sources. Out-of-the box SaaS content packs that deliver a curated experience with pre-built dashboards to get you up and running quickly. We have hundreds of ISVs building content packs to cater to the needs of millions of Power BI users. Unmatched, unique ways for users to experience their data with speed and agility: Live dashboards that maintain a real-time pulse on the business and provide critical insights. Natural language query that enables users to simply and intuitively ask questions of their data, including through Cortana. Custom visuals that bring data to life and surface intelligence hidden in the sea of data, with our community leveraging the Power BI visualization stack to create new ways to visualize data in a way that makes more sense. (Now available in the Office store.) Integration of Power BI with the Microsoft stack. Power BI is part of larger ecosystem that integrates with services like Microsoft Teams, Office 365, and Dynamics 365. These services are aware of Power BI, are wired to Power BI, and enable you to use Power BI in the context of your work. Anywhere access to insights. Whether in the office or on-the-go, Power BI provides anywhere access to insights with dashboards accessible via the desktop, on the web, or across mobile devices. Inside Excel, embedded—we have hundreds of ISVs embedding Power BI in their offerings.

Microsoft Power BI Technical Overview

David J Rosenthal

Introduction to Data Analytics

Dr. C.V. Suresh Babu

Ha100 notes units 1 and 2 sp08

Duskydope Rao

PowerApps and Flow are services for building and using custom business apps that connect to your data and work across the web and mobile - without the time and expense of custom software development. However, there are scenarios where the need to customize and develop extra features for workflows are vital, in order to take workflows to the next level. This session demonstrates key advanced scenarios addressing the next level workflows using PowerApps and Flow, featuring Azure Api Apps and Azure Functions.

SPS Brno 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Ahmad Najjar

SharePoint 2010: Insights into BI

J.D. Wade

The slide deck for the Power Platform Presentation in SQL Saturday Redmond 2019. We have reviewed the power Platform Components, why is it better together and how to make it happen. During the demo all the options of implementation between the Power Apps and PowerBI were demonstrated. Including the data visualization changes with new data feed. Use some of the following ideas in your organization and POC's for more complex implementations.

SQL Saturday Redmond The Power Platform

Berkovich Consulting

The Briefing Room with Dr. Robin Bloor and WhereScape Live Webcast on April 1, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7b23b14b532bd7be60a70f6bd5209f03 In the Big Data shuffle, everyone is looking at Hadoop as “the answer” to collect interesting data from a new set of sources. While Hadoop has given organizations the power to gather more information assets than ever before, the question still looms: which data, regardless of source, structure, volume and all the rest, are significant for affecting business value – and how do we harness it? One effective approach is to bolster the data warehouse environment with a solution capable of integrating all the data sources, including Hadoop, and automating delivery of key information into the rights hands. Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor as he explains how a rapidly changing information landscape impacts data management. He will be briefed by Mark Budzinski of WhereScape, who will tout his company’s data warehouse automation solutions. Budzinski will discuss how automation can be the cornerstone for closing the gap between those responsible for data management and the people driving business decisions. Visit InsideAnlaysis.com for more information.

The Right Data Warehouse: Automation Now, Business Value Thereafter

Inside Analysis

SPS Dubai 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Ahmad Najjar

Projects are dynamic. Staying ahead of the curve means having real-time access to your project data and it’s leading indicators for actionable intelligence. A real-time, flexible platform Tivitie Insights service enables organizations to visualize critical Microsoft Project Online data in real-time, spot trends as they occur and produce actionable facts for better decision making in a fraction of the time.

Project Insights for Data Driven Decisions

Michelle Manimtim

Loading text data from SAP source systems

Marcelo Honores

In this session, you learn about the latest and hottest features of Amazon Redshift. Join Vidhya Srinivasan, General Manager of Amazon Redshift, to take a deep dive into the architecture and inner workings of Amazon Redshift. You discover how the recent availability, performance, and manageability improvements we’ve made can significantly enhance your end user experience. You also get a glimpse of what we are working on and our plans for the future.

AWS re:Invent 2016: What’s New with Amazon Redshift (BDA304)

Amazon Web Services

SPS London 2017 - Building applications with PowerApps, Microsoft flow and Of...

Ahmad Najjar

Similaire à Let's Build a Service Oriented Data Pipeline! (20)

SPS Warsaw 2017 - Building applications with PowerApps, Microsoft flow and Of...

SPS Brno 2017 - Go with the Microsoft flow

Introduction to Microsoft Power BI

Jira - Solving Reporting Problems using eazyBI

Synapse NanoApps

Microsoft Business Intelligence - Practical Approach & Overview

Intro to PowerApps and Flow

Automating your tasks with microsoft flow

Microsoft Power BI Technical Overview

Introduction to Data Analytics

Ha100 notes units 1 and 2 sp08

SPS Brno 2017 - PowerApps & Microsoft Flow: Advanced scenarios

SharePoint 2010: Insights into BI

SQL Saturday Redmond The Power Platform

The Right Data Warehouse: Automation Now, Business Value Thereafter

SPS Dubai 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Project Insights for Data Driven Decisions

Loading text data from SAP source systems

AWS re:Invent 2016: What’s New with Amazon Redshift (BDA304)

SPS London 2017 - Building applications with PowerApps, Microsoft flow and Of...

Dernier

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

A Beginners Guide to Building a RAG App Using Open Source Milvus

Zilliz

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

AXA XL - Insurer Innovation Award Americas 2024

The Digital Insurer

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Navi Mumbai Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Navi Mumbai Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Navi Mumbai Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Dernier (20)

AWS Community Day CPH - Three problems of Terraform

Apidays New York 2024 - The value of a flexible API Management solution for O...

A Beginners Guide to Building a RAG App Using Open Source Milvus

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

AXA XL - Insurer Innovation Award Americas 2024

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

A Year of the Servo Reboot: Where Are We Now?

Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Exploring the Future Potential of AI-Enabled Smartphone Processors

Data Cloud, More than a CDP by Matt Robison

presentation ICT roal in 21st century education

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Boost Fertility New Invention Ups Success Rates.pdf

Why Teams call analytics are critical to your entire business

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Ransomware_Q4_2023. The report. [EN].pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

Let's Build a Service Oriented Data Pipeline!

1. Let’s Build a Service Oriented Data Pipeline! June 2016 Software Developer | Hootsuite Yasha Podeswa

2. Before: Oceanographer Me!

3. Now: Software Developer at Hootsuite Me!

4. Introduce a problem that requires a new data pipeline Design it in a service oriented style Build it on stage! This Talk

5. Passive Aggressive Inc. just cancelled their subscription! Desperate Dan in trouble! The Problem

6. Want to Build a Tool Like This

7. Want to Build a Tool Like This

8. Want to Build a Tool Like This

10. What We’re Starting With

11. What We’re Starting With Things Users Did

12. What We’re Starting With Things Organizations Did

13. What We’re Starting With Crap

14. High Level Plan JSON files Calculate stats about organizations DB

15. High Level Plan JSON files Calculate stats about organizations DB Extract Transform Load

16. High Level Plan JSON files Calculate stats about organizations DB Extract Transform Load

17. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization

18. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization Useful for lots of things!

19. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization Shouldn’t run until dependent job done

20. Need a “Service” Communication and Orchestration Layer!

21.

22. Let’s build it!

23. First App Event Cleaning and Loading Read logs from S3, clean and sort into different types of events, load into data warehouse Vanilla Scala app AWS Lambda

24. Second App Organization Stat Calculation Read cleaned/sorted events from data warehouse, calculate stats about organization, load stats to data warehouse Vanilla Scala app AWS Lambda

25. Third App Airflow Hook up the Lambda apps in a dependency graph ● Scheduling ● Retries ● Monitoring

26. Steal my code! https://github.com/yashap/etl-load-events https://github.com/yashap/etl-organization-stats https://github.com/yashap/airflow

27. Questions?

Let's Build a Service Oriented Data Pipeline!

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (14)

Similaire à Let's Build a Service Oriented Data Pipeline!

Similaire à Let's Build a Service Oriented Data Pipeline! (20)

Dernier

Dernier (20)

Let's Build a Service Oriented Data Pipeline!