Open-source vs. public cloud in the Big Data landscape. Friends or Foes?

•

2 j'aime•480 vues

If you want to stay up to date, subscribe to our newsletter here: https://bit.ly/3tiw1I8 A presentation about the strong competition between open-source vendors and public cloud providers in the Big Data landscape.

Données & analyses

Open-source vs.
public cloud.
Friends or Foes?
Adam Kawa, GetInData

Some of dilemmas
● Hive MR/Tez/LLAP, Impala, Presto, Spark SQL, Drill
● Oozie, Azkaban, Luigi, Airﬂow
● Kafka, Flume
● Storm, Spark Streaming, Samza, Flink
● Cloudera, Hortonworks, MapR
● GCP, AWS, Azure

With or without cloud
● Amazon Elastic MapReduce launched in April 2009
● Most of the companies started on-premise
○ Y!, FB, Twitter, LI, Allegro, ING
● Several companies were happy with the cloud
○ Netﬂix, Base CRM, Wikia
● A few companies went back and forth
○ Spotify, Foursquare, Synerise

Open-source
vs. public cloud
2016 - 2019.

Dark clouds over open-source
1. Provides proprietary & vendor lock-in products
○ Kinesis, PubSub, Redshift, BigQuery

Dark clouds over open-source
1. Provides proprietary & vendor lock-in products
○ Kinesis, PubSub, Redshift, BigQuery
2. Strip-mines open-source projects
○ Amazon Managed Streaming for Kafka (MSK)
○ Amazon Elasticsearch Service
○ Cloud Composer (Airﬂow), Cloud Datalab (Jupyter)

Run software in
containers on
premises and in the
cloud

Open-source responses
● Self-defense licenses
○ Conﬂuent (Kafka), MongoDB, Redis
● Own cloud-based products
○ Databricks (Spark), Conﬂuent (Kafka), Cloudera
(Hadoop)
● Cloud-ﬁrst approach
○ That code is released ﬁrst to their cloud oﬀering

Current trends
1. More companies start using or evaluating the
cloud
2. Companies usually avoid or sometimes accept
vendor lock-in
3. Open-source projects become cloud-ﬁrst and
cloud-ready
4. Kubernetes provides a portability layer

Native open-source
services on Google
Cloud Platform
● a seamless sign-up
experience
● an integrated billing
● the 1st
line support
provided directly by
Google Cloud

Win! Win! Win!
Open-source
communities
and vendors
Cloud providers Customers

Mix of
open-source
and cloud
● Kafka
● Google Cloud Storage +
DataProc (Spark)
● Hive, Presto & BigQuery
● Tableau

Hybrid
environment
(test and prod)
● Real-time user journey
analytics and interactions
○ Kafka, Flink, Hive,
Docker
● Development in the cloud
● Production on=premise

Open-source vs. public cloud in the Big Data landscape. Friends or Foes?

Contenu connexe

Tendances

Serverless Swift for Mobile Developers

All Things Open

Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience Lond...

InfluxData

Come and join us for an (almost) no-slides session around the terrific trio of Debezium, Apache Kafka Streams, and Knative Eventing! Leveraging Apache Kafka as the de-facto standard for event-driven data pipelines, these open-source technologies allow you to ingest data changes from relational and NoSQL databases, process and enrich them, and consume them serverless-style. In a live demo, you’ll see how Debezium, Apache Kafka, Quarkus, and Knative are the dream-team for building serverless, cloud-native stream processing pipelines. You will learn: How to stream change events out of your database using Debezium How to use the Quarkus extension for Kafka Streams to build cloud-native stream processing applications, running either on the JVM or GraalVM How to consume and distribute Kafka messages with Knative Eventing, allowing you to manage modern serverless workloads on Kubernetes.

Serverless stream processing of Debezium data change events with Knative | De...

Red Hat Developers

DevOps project sprawl is real! Large organizations with many teams need to support a variety of configurations from infrastructure governance to domain-specific app deployments, all while enforcing good security practices like least privilege for each team. Maintaining these controls by hand leads to complexity, stagnation, and insecure shortcuts. In this session, you'll learn how Terraform can automate this configuration--using Terraform--and make doing the right thing easy!

Advanced dev ops governance with terraform

James Counts

The goal of Serverless is to focus on writing the code that delivers business value and offload everything else to your trusted partners (like Cloud providers or SaaS vendors). You want to iterate quickly and today’s code quickly becomes tomorrow’s technical debt. In this talk we will show why Serverless adoption increases the developer productivity and how to measure it. We will also go through AWS Serverless architectures where you only glue together different Serverless managed services relying solely on configuration, minimizing the amount of the code written.

Measure and Increase Developer Productivity with Help of Serverless at JCON 2...

Vadym Kazulkin

Datadog- Monitoring In Motion

Cloud Native Apps SF

No matter what type of IoT devices you have, or what your use case is for them, you’re going to end up producing a lot of time series data. What you use to handle it is going to be as important to your success as the code you write yourself. This talk will evaluate the available open source tools for the collection, activation, transmission and visualization of time series data on the IoT Edge, and demonstrate how to use them, together with InfluxDB, to solve various use cases for the Internet of Things.

Time Series Tech Stack for the IoT Edge

InfluxData

Metaflow (Ville Tuulos) Data scientists at Netflix are expected to develop and operate large machine learning workflows autonomously. However, we do not expect that all our scientists are deeply experienced with distributed systems and data engineering. Metaflow was created to make it delightfully easy to build and operate ML workflows in the cloud using idiomatic Python and off-the-shelf ML libraries, covering the whole lifecycle of an ML project from prototype to production. Polynote (Jeremy Smith) Polynote is a new notebook tool we created from scratch to address some of the pain points we've run into while using Scala in machine-learning notebooks at Netflix. It provides essential code editing features other tools lack like interactive auto-completes, support for mixing multiple languages and sharing data between them within a single notebook, and encourages reproducible notebooks with its immutable data model. Papermill (Matthew Seal) Nteract is an open source organization under which there are several libraries and applications that Netflix and many other companies and individuals contribute to. One of these libraries is Papermill, a library used to programmatically parameterize and execute Jupyter Notebooks. Papermill provides a CLI and Python interface that we'll explore during the session to see how it can be used and what value it adds. Using this pattern we'll also briefly talk about how we've integrated papermill at Netflix and how it interfaces with other Jupyter and nteract services.

Season 7 Episode 1 - Tools for Data Scientists

aspyker

http://flink-forward.org/kb_sessions/flink-and-beam-current-state-roadmap/ It is no secret that the Dataflow model, which evolved from Google’s MapReduce, Flume, and MillWheel, has been a major influence to Apache Flink’s streaming API. The essentials of this model are captured in Apache Beam. Beam provides the Dataflow API with the option to deploy to various backends (e.g. Flink, Spark). In this talk we will examine the current state of the Flink Runner. Beam’s Runners manage the translation of the Beam API into the backend API. The Beam project itself has made an effort to summarize the capabilities of each Runner to provide an overview of the supported API concepts. From all open sources backends, Flink is currently the Runner which supports the most features. We will look at the supported Beam features and their counterpart in Flink. Further, we will look at potential improvements and upcoming features of the Flink Runner.

Maximilian Michels - Flink and Beam

Flink Forward

CS80A Foothill College Open Source Talk

aspyker

We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Red Hat Developers

Persist your data in an ephemeral k8 ecosystem

LibbySchulze

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Complex computational workloads in Python are a common sight these days, especially in the context of processing large and complex datasets. Battle-hardened modules such as Numpy, Pandas, and Scikit-Learn can perform low-level tasks, while tools like Dask makes it easy to parallelize these workloads across distributed computational environments. Meanwhile, Argo Workflows offers a Kubernetes-native solution to provisioning cloud resources in Kubernetes and triggering workflows on a regular schedule. Being Kubernetes-native, Argo Workflows also meshes nicely with other Kubernetes tools. This talk discusses the combination of these two worlds by showcasing a set-up for Argo-managed workflows which schedule and automatically scale-out Dask-powered data pipelines in Python. BIO Former academic in the field of renewable energy simulation and energy systems analysis. Currently responsible for architecting and maintaining the cloud- and data strategy at ACCURE Battery Intelligence KEY TAKE-AWAYS FROM THE TALK Argo Workflows + Dask is a nice combination for data-processing pipelines. There are a a few "gotchyas" to be on the look-out for, but in nevertheless this is still a generally-applicable and powerful combination. https://github.com/sevberg

Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows

DoKC

NetflixOSS and ZeroToDocker Talk

aspyker

The service mesh management plane

LibbySchulze

In this episode, we will focus on continuous delivery and how Netflix uses Spinnaker and Kayenta to safely deliver changes to the cloud and beyond. Kayenta is a platform for Automated Canary Analysis (ACA). It is used by Spinnaker to enable automated canary deployments. We will also discuss how Spinnaker is used at Netflix to deploy targets beyond cloud VMs and containers --- batch jobs, CDNs, fast properties and Open Connect appliances.

NetflixOSS Meetup S6E2 - Spinnaker, Kayenta

aspyker

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Timothy Spann

Netflix Cloud Architecture and Open Source

aspyker

Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen

InfluxData

Cloud native policy enforcement with Open Policy Agent

LibbySchulze

Tendances (20)

Serverless Swift for Mobile Developers

Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience Lond...

Serverless stream processing of Debezium data change events with Knative | De...

Advanced dev ops governance with terraform

Measure and Increase Developer Productivity with Help of Serverless at JCON 2...

Datadog- Monitoring In Motion

Time Series Tech Stack for the IoT Edge

Season 7 Episode 1 - Tools for Data Scientists

Maximilian Michels - Flink and Beam

CS80A Foothill College Open Source Talk

A Microservices approach with Cassandra and Quarkus | DevNation Tech Talk

Persist your data in an ephemeral k8 ecosystem

Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows

NetflixOSS and ZeroToDocker Talk

The service mesh management plane

NetflixOSS Meetup S6E2 - Spinnaker, Kayenta

Using the FLaNK Stack for edge ai (apache mxnet, apache flink, apache nifi, a...

Netflix Cloud Architecture and Open Source

Container Monitoring Best Practices Using AWS and InfluxData by Gunnar Aasen

Cloud native policy enforcement with Open Policy Agent

Similaire à Open-source vs. public cloud in the Big Data landscape. Friends or Foes?

Cloud computing

Richard Claassens CIPPE

Presented on April 27th, 2013 at LinuxFest NW Imagine it’s eight o’clock on a Thursday morning and you awake to see a bulldozer out your window ready to plow over your data center. Normally you may wish to consult the Encyclopedia Galáctica to discern the best course of action but your copy is likely out of date. And while the Hitchhiker’s Guide to the Galaxy (HHGTTG) is a wholly remarkable book it doesn’t cover the nuances of cloud computing. That’s why you need the Hitchhiker’s Guide to Cloud Computing (HHGTCC) or at least to attend this talk understand the state of open source cloud computing. Specifically this talk will cover infrastructure-as-a-service, platform-as-a-service and developments in big data and how to more effectively take advantage of these technologies using open source software. Technologies that will be covered in this talk include Apache CloudStack, Chef, CloudFoundry, NoSQL, OpenStack, Puppet and many more. Specific topics for discussion will include: Infrastructure-as-a-Service - The Systems Cloud - Get a comparision of the open source cloud platforms including OpenStack, Apache CloudStack, Eucalyptus, OpenNebula Platform-as-a-Service - The Developers Cloud - Find out what tools are availble to build portable auto-scaling applications including CloudFoundry, OpenShift, Stackato and more. Data-as-a-Service - The Analytics Cloud - Want to figure out the who, what , where , when and why of big data ? You get an overview of open source NoSQL databases and technologies like MapReduce to help crunch massive data sets in the cloud. Finally you'll get a overview of the tools that can help you really take advantage of the cloud? Want to auto-scale virtual machiens to serve millions of web pages or want to automate the configuration of cloud computing environments. You'll learn how to combine these tools to provide continous deployment systems that will help you earn DevOps cred in any data center. [Finally, for those of you that are Douglas Adams fans please accept the deepest apologies for bad analogies to the HHGTTG.]

LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing

Mark Hinkle

Anil Inamdar, VP & Head of Data Solutions, Instaclustr Most organizations considering open source and open core cloud technologies as part of their all-important data stack understand they need to rigorously evaluate the software's licensing terms and gauge the long-term health of its community and ecosystem. What still happens less frequently ' but is just as crucial to these risk assessments ' is developing a thorough understanding of the business models governing the commercial organizations attached to each data-layer technology being considered. You must discern the underlying motivations of the vendors or technology providers you depend on to deliver or support open source data-layer software (as well as those vendors with strong influence over its development and maintenance). By acutely understanding these incentives, you can identify if, where, and how they may map to possible risks to your enterprise's adoption and ongoing open source implementation. Don't limit the assessment to licenses and community health -- although both are still very key variables. This session will discuss specifics on what you need to look for and consider when vetting open source data technologies in the cloud as offered by: -- Businesses using OSS as the foundation of their own intellectual property -- Businesses that maintain total control offer the OSS they offer -- Major cloud providers

Data Con LA 2022-Open Source or Open Core in Your Data Layer? What Needs to B...

Data Con LA

Cf intro for spring devs

Eric Bottard

Software Strategy

Alec Shutze

Self-Service Supercomputing

Amazon Web Services

Imagine it's eight o'clock on a Thursday morning and you awake to see a bulldozer out your window ready to plow over your data center. Normally you may wish to consult the Encyclopedia GalÃ¡ctica to discern the best course of action but your copy is likely out of date. And while the Hitchhiker's Guide to the Galaxy (HHGTTG) is a wholly remarkable book it doesn't cover the nuances of cloud computing. That's why you need the Hitchhiker's Guide to Cloud Computing (HHGTCC) or at least to attend this talk understand the state of open source cloud computing. Specifically this talk will cover infrastructure-as-a-service, platform-as-a-service and developments in big data and how to more effectively take advantage of these technologies using open source software. Technologies that will be covered in this talk include Apache CloudStack, Chef, CloudFoundry, NoSQL, OpenStack, Puppet and many more.

Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud

Mark Hinkle

Watch this talk here: https://www.confluent.io/online-talks/bridge-to-cloud-apache-kafka-migrate-aws Speakers: Priya Shivakumar, Director of Product, Confluent + Konstantine Karantasis, Software Engineer, Confluent + Rohit Pujari, Partner Solutions Architect, AWS Most companies start their cloud journey with a new use case, or a new application. Sometimes these applications can run independently in the cloud, but often times they need data from the on premises datacenter. Existing applications will slowly migrate, but will need a strategy and the technology to enable a multi-year migration. In this session, we will share how companies around the world are using Confluent Cloud, a fully managed Apache Kafka service, to migrate to AWS. By implementing a central-pipeline architecture using Apache Kafka to sync on-prem and cloud deployments, companies can accelerate migration times and reduce costs. In this online talk we will cover: •How to take the first step in migrating to AWS •How to reliably sync your on premises applications using a persistent bridge to cloud •Learn how Confluent Cloud can make this daunting task simple, reliable and performant •See a demo of the hybrid-cloud and multi-region deployment of Apache Kafka

Bridge to Cloud: Using Apache Kafka to Migrate to AWS

confluent

Python on AWS Lambda

Amazon Web Services

Drupal has been a consistent leader in the Gartner Magic Quadrant for Web Content Management. However, enterprises leveraging Drupal have traditionally relied on PaaS providers for their hosting, scaling and lifecycle management. And that usually leads to enterprise applications being locked-in with a particular cloud or vendor. As container and container orchestration technologies disrupt the cloud and platform landscape, there’s a clear way to avoid this state of affairs. In this webinar, we discuss why it's important to build a cloud-native Drupal platform, and exactly how to do that. Join the webinar to understand how you can avoid vendor lock-in, and create a secure platform to manage, operate and scale your Drupal applications in a multi-cloud portable manner. Key Takeaways: - Why you need a cloud-native Drupal platform and how to build one - How to craft an idiomatic development workflow - Understanding infrastructure and cloud engineering - under the hood - Demystifying the art and science of Docker and Kubernetes: deep dive into scaling the LAMP stack - Exploring cost optimization and cloud governance - Understand portability of applications - A hands-on demo of how the platform works

[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...

Srijan Technologies

Avoiding cloud lock-in

Sebastien Goasguen

Cloud Foundry Introduction and Overview

Andy Piper

Cloud computing overview

daklug

Keeping your options open

Doug Tidwell

Hybrid Cloud Service on the ThinkFree Mobile Android Platform

Benedict Ji

Open source and cloud computing are two terms that everyone seems to be talking about. Powerhouses on their own, when paired together open source and cloud computing can create a developer’s dream scenario. In this session, Bret Piatt, technical alliances at Rackspace Hosting will discuss the history of open source software development and the spread of open source across the internet. Cloud computing providers are now incorporating open source into their business models through open APIs and contributions to various open source projects such as Cassandra and Drizzle, and Bret will discuss these developments while taking a close look at the intersection of cloud computing and open source to cover: How cloud computing is changing open source How cloud computing can benefit from open source How open source will lead the interoperability push How the success of cloud is tied to mass adoption that requires interoperability

Future of Open Source in a Cloudy World

Bret Piatt

20091012 Command Your Data

Open-Xchange

Cloud computing.pptx

ProfRajivKumarShobhi

Building Real-time Travel Alerts In this session, we will walk through how to build a complete streaming application to send alerts based on travel advisories from public data. We will also join in other data sources of relevance and push out alerts. We will show you how to build this streaming application with Apache NiFi, Apache Kafka, and Apache Flink and show you when/why/how, and what to build to maximize performance, productivity, and ease of development. Let's get streaming. Apache Flink Apache Kafka Apache NiFi FLaNK Stack Tim Spann Big Data Conference Europe 2023

Building Real-Time Travel Alerts

Timothy Spann

Internet6: A Digital Game Changer

CARLOS RALLI-UCENDO

Similaire à Open-source vs. public cloud in the Big Data landscape. Friends or Foes? (20)

Cloud computing

LinuxFest NW 2013: Hitchhiker's Guide to Open Source Cloud Computing

Data Con LA 2022-Open Source or Open Core in Your Data Layer? What Needs to B...

Cf intro for spring devs

Software Strategy

Self-Service Supercomputing

Linux Foundation Collaboration Summit: Hitchhiker's Guide to the Cloud

Bridge to Cloud: Using Apache Kafka to Migrate to AWS

Python on AWS Lambda

[Srijan Wednesday Webinars] How to Build a Cloud Native Platform for Enterpri...

Avoiding cloud lock-in

Cloud Foundry Introduction and Overview

Cloud computing overview

Keeping your options open

Hybrid Cloud Service on the ThinkFree Mobile Android Platform

Future of Open Source in a Cloudy World

20091012 Command Your Data

Cloud computing.pptx

Building Real-Time Travel Alerts

Internet6: A Digital Game Changer

Plus de GetInData

How do we work with our customers ? How does it look? What do the meetings look like ? How do we structure the cooperation? Who does what and when ? We receive these kinds of questions quite often. They are very important questions as the customer should know the details before we start the project and it’s important for GetInData to be transparent on this so the client is well informed. During the webinar our Project Lead, Rafał Zalewski talked about Scrum Framework we use in cooperation with our customers. Watch here: https://www.youtube.com/watch?v=uOWrgcaKwWo&t=32s Speaker: Rafał Zalewski, GetInData: https://www.linkedin.com/in/rafalzalewski/ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

How do we work with customers on Big Data / ML / Analytics Projects using Scr...

GetInData

Watch video here: https://youtu.be/sfowpU90zFM Piotr's presentation about GetInData’s Data-Driven Fast Track, the 3-step framework for data transformation. You will learn: ➡ How to assess how data-driven your company is ➡ How to generate ideas for new initiatives to push your company towards better decisions ➡ How to think about implementing these initiatives to increase your chances of success If you miss it live don't despair. Watch the video and feel free to diagnose your company by filling out the survey prepared by our team here: https://bit.ly/3fKcRrb! After completing the survey, you will receive a tailored summary report with insights from one of our experts. Below you'll find links to all the materials mentioned in the workshop needed for exercises. LINKS TO MATERIALS ABOUT DATA-DRIVEN: Data-driven fast-track: 3 steps to make your company more data-driven: https://getindata.com/blog/data-drive... Is my company data-driven? Here’s how you can find out: https://getindata.com/blog/is-my-comp... If you: ➡ have questions about webinar topic, ➡ want to talk about your data-driven transformation, ➡ want to become more data-driven and you need consultations, don't hesitate to write to us: hello@getindata.com Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Data-Driven Fast Track: Introduction to data-drivenness with Piotr Menclewicz

GetInData

If you want to stay up to date, subscribe to our newsletter here: https://bit.ly/3tiw1I8 Presentation from the performance given by Piotr Chaberski and Adrian Dembek the Data Science Summit ML Edition. Authors: Piotr Chaberski, Adrian Dembek Linkedin: https://www.linkedin.com/in/piotrchaberski/ https://www.linkedin.com/in/adriandembek/ ___ Company: Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

How NOT to win a Kaggle competition

GetInData

Speaker: Rafał Zalewski, GetInData: https://www.linkedin.com/in/rafalzalewski/ Abstract: To become good Developer in Scrum Team you need to understand not only Scrum Events but also Scrum fundaments like Scrum Pillars and Scrum Values. In this presentation you will learn and understand the mindset expectation from you as Developer in Scrum Team. You will also learn how Scrum mindset helps to achieve better development results. ____ Company: Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

How to become good Developer in Scrum Team?

GetInData

If you want to stay up to date, subscribe to our newsletter here: https://bit.ly/3tiw1I8 Presentation from the performance given by Paweł during the Airflow Summit 2022. Author: Paweł Leszczyński Linkedin: https://www.linkedin.com/in/pawel-leszczynski/ ___ Company: Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

OpenLineage & Airflow - data lineage has never been easier

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog Building your own platform is often ostracized these days. Everyone is encouraged to reuse existing solutions for known reasons. But using a ready-made platform / tool should not be a mindless process. Reusability is an art. During this presentation, you will learn why we decided to build our own MLOps platform while not re-inventing the wheel by using ready-made components with a touch of custom components. What are the benefits of this, but also what limitations and hurdles we have encountered. We hope that our experience will help you make the right decisions in your projects. Sometimes, maybe more risky ones.

Benefits of a Homemade ML Platform

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog This workshop focuses on creating a data streaming platform from scratch using an empty Kubernetes (or even Minikube) cluster. During the workshop, we go through the installation process, deploy the basic components for the platform, start Apache Flink, and monitor the process, using SQL to query available data. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewan...

GetInData

Check out more about this presentation here: https://www.youtube.com/watch?v=nSsssYHiylQ&t=17s Presentation from the performance given by our team during the NSML Summit. Authors: Krzysztof Zarzycki, Marek Wiewiórka Linkedin: https://www.linkedin.com/in/kzarzycki/ https://www.linkedin.com/in/marekwiewiorka/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

MLOps implemented - how we combine the cloud & open-source to boost data scie...

GetInData

Feast + Amundsen Integration - Mariusz Strzelecki, GetInData

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog More and more services are running in Kubernetes so it means that we can migrate our current data pipelines to the new environment. In case of Flink we have multiple ways to do real-time data streaming: use Lyft or GCP operator, go with official deployment and customize it or choose the Ververica Platform or create something on your own. The presentation shows how to choose the right solution for technical requirements and business needs to run Flink in Kubernetes at great scale with no issues. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Kubernetes and real-time analytics - how to connect these two worlds with Apa...

GetInData

If you want to stay up to date, subscribe to our newsletter here: https://bit.ly/3tiw1I8 Get more info here: https://getindata.com/blog/6-big-data-trends-2021-bigdata-blog/ Author: Krzysztof Zarzycki Linkedin: https://www.linkedin.com/in/kzarzycki/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Big data trends - Krzysztof Zarzycki, GetInData

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog The talk is focused on administration, development and monitoring platform with Apache Spark, Apache Flink and Kubeflow in which the monitoring stack is based on Prometheus stack. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...

GetInData

Check out more about this presentation here: https://www.youtube.com/watch?v=eqNToHn4yB0 The webinar was organized by GetinData on 2020. During the webinar we explaned what does it mean to build a data-driven company. Watch more here: https://www.youtube.com/watch?v=eqNToHn4yB0 Speaker: Rafał Małanij ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Analytics 101 - How to build a data-driven organisation? - Rafał Małanij, Get...

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog The webinar was organized by GetinData on 2020. During the webinar we explaned the concept of monitoring and observability with focus on data analytics platforms. Watch more here: https://www.youtube.com/watch?v=qSOlEN5XBQc Whitepaper - Monitoring ang Observability for Data Platform: https://getindata.com/blog/white-paper-big-data-monitoring-observability-data-platform/ Speaker: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Monitoring in Big Data Platform - Albert Lewandowski, GetInData

GetInData

If you want to learn more about it, check out our webinar here: https://www.youtube.com/watch?v=EfGPY_NyYQ8&t=77s The webinar was organized by GetinData on 2020. During the webinar, we shared our lessons learnt from building and running stream processing platform in production for over 2 years. Watch more here: https://www.youtube.com/watch?v=EfGPY_NyYQ8 Author: Krzysztof Zarzycki Linkedin: https://www.linkedin.com/in/kzarzycki/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Complex event processing platform handling millions of users - Krzysztof Zarz...

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog Nowadays, one tweet can have impact on the value of the company or cryptocurrency. It becomes important for companies to be able to know everything what's happening in the market, especially for startups or when entering the new market. The presentation is about presenting the complex platform used for creating and verifying the strategy for a startup from the Wellbeing market. We go through web scraping-based data ingestion to ElasticSearch, NLP pipelines to understand what people write and what is the possible future of each market predicted by PySpark job. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Predicting Startup Market Trends based on the news and social media - Albert ...

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog Currently there are more and more created videos distributed via multiple social media channels. It becomes more and more important to monitor all of them by companies to verify their customers' feedback, reviews, opinions. During the talk, we talk about extracting text from videos, analyzing language and prepare robust, scalable infrastructure for it. The idea behind platform is about having the mix between managed and self-managed service for Big Data processing. The keynote shows the case study of the MVP of the platform for marketing companies. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

NLP for videos: Understanding customers' feelings in videos - Albert Lewandow...

GetInData

If you want to learn more about it watch our video: https://www.youtube.com/watch?v=WjAY9mnqD4s The webinar was organized by GetinData on 2020, June 9th. We discussed various approaches one can take to migrate the infrastructure, pointing out their pros and cons. The webinar was based on our real-life, production-grade experiences, however, the strategies and insights can be easily applied to all kinds of IT business. Watch our webinar here: https://www.youtube.com/watch?v=WjAY9mnqD4s&t=1s Speaker: Mateusz Pytel Linkedin: https://www.linkedin.com/in/em-pe/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Strategies for on premise to Google Cloud migration - Mateusz Pytel, GetInData

GetInData

Did you like it? Check out our blog to stay up to date: https://getindata.com/blog Satellite data is not well-known although it provides a lot useful information for different sectors like agriculture, industry or military.During the workshops participants will learn how to get the satellite imageries, process them by using Python and PySpark and what are the most common use cases. There will be a knowledge sharing session about issues with satellite data and how these difficulties can be overcome. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ _____ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Monitoring environment based on satellite data with Python and PySpark - Albe...

GetInData

If you want to learn more, check out our performance given by our team during the Big Data Technology Warsaw Summit: https://www.youtube.com/watch?v=gdinM_9R2GA&t=22s Operationalizing Machine Learning operations (features delivery, model training, deployment and serving) is nowadays one of the most challenging areas in fast-growing data-driven companies. Variety of open source components (Kubeflow, Mlflow, Kedro to name a few) and set of specialized managed services provided by every major cloud provider drive solution architects nuts. In GetInData we have a solution for it - we used to call it GetInData MLOps Platform. Set of reusable components, following the Unix toolset pattern ("do one thing and be best at it") and portable to any environment. Also - thanks to loose coupling - adjustable to current and future clients' ML-related challenges, like a candy shop where the first person needs super-fast online predictions, the second one requires robust hyperparameter tuning for best possible models and the third person aims for scalable collaboration on features extraction within many data science teams. During the presentation we will show you two components we're really excited for - Kedro-Kubeflow integration and Feast-based feature store - how we implement these and what clients' use them for. Welcome to our MLOps candy shop that no pandemic can close Watch our webinar here: https://www.youtube.com/watch?v=gdinM_9R2GA&t=10s Speakers: Mateusz Pytel - Google Certified Professional: https://www.linkedin.com/in/em-pe/ Mariusz Strzelecki - Senior Machine Learning Engineer: https://www.linkedin.com/in/mariusz-strzelecki/ ___ Company: Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Welcome to MLOps candy shop and choose your flavour! - Mateusz Pytel & Marius...

GetInData

Plus de GetInData (20)