The Critical Missing Component in the Production ML Stack

•

0 j'aime•74 vues

The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system? In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.

Données & analyses

The critical missing
component in the
production ML stack
Alessya Visnjic

CEO, WhyLabs.ai

Agenda:

▪ ML Stack: what is missing?

▪ How to design data logging

▪ whylogs: open standard for data logging

▪ Use cases

▪ Q&A

ML models are full of
surprises…
… every surprise will
launch a debugging
expedition!

ML models routinely struggle in the wild…

ML Stack: moving massive volumes of data

Test

Monitor

Debug

Document
How do you data?

ML Stack is missing data & metadata logging
Log metadata &
statistical properties
of data

Good data log
should capture:
Metadata

Counts

Statistics

Distributions

Stratified samples

Key properties

of a data log:
Lightweight

Portable

Mergeable

Configurable

Close to code

whylogs:

logging for
the ML stack bit.ly/whylogs

whylogs: a standard format for
representing a snapshot of data

bit.ly/whylogs

Feature name count max min stddev nunique null_count quantile_0.0000 … quantile_1.0000
chlorides 1199.0 0.611 0.012 0.044 134.0 0.0 0.012 … 0.611
quality 1199 8.000 3.000 0.785 6.0 0.0 3.000 … 8.000
alcohol 1199 14.900 8.400 1.060 65.0 0.0 8.400 … 14.900
density 1199 1.004 0.997 0.001 390.0 0.0 0.990 … 1.004
pH 1199 4.010 2.890 0.153 82.0 0.0 2.890 … 4.010
Log rich statistics for each feature
Each data log captures summary statistics, counters, distributions, metadata and custom metrics
Sample of a flattened data log captured by whylogs on the Wine Quality dataset

Track data statistics across batches
Distribution plot for one of the columns in the model input, collected at inference time

Distribution of “free sulfur dioxide” feature over 20 inference batches of the Wine Quality model

Dataset Size # of entries # of features Memory consumption Output size
Lending Club 1.6G 2.2M 151 14MB 7.4MB
NYC Tickets 1.9G 10.8 43 14MB 2.3MB
Pain pills 75GB 178M 42 15MB 2MB
Run data logging without overhead
Using streaming algorithms to capture data statistics, whylogs ensures a constant memory footprint,
scales with the number of features in the dataframe, and outputs lightweight log files (json, protobuf, etc).
Sample of whylogs benchmarks on public datasets

Whylogs profiles 100% of the data to accurately capture distributions. Calculating distributions from randomly sampled data is significantly
less accurate. The chart presents median error for distributions estimated with whylogs vs. random sampling techniques.
0
0.1
0.2
0.2
0.3
0.4
Normal Normal discrete Normal outlier Uniform discrete Uniform Pareto
Profiling Sampling
Capture accurate data distributions

Test

Monitor

Debug

Document
You can
data with

whylogs!

Whylogs captures mergeable histograms for each feature. To catch distribution drift,
continuously compare training distribution of a feature to the serving distribution.
0
200
400
600
800
1000
Training Serving
Use case: training-serving distribution drift

Logging enables all key MLOps activities
Once data is logged systematically, whylogs outputs can be used to test, monitor, and debug data.
Use whylogs at any point of the ML stack and through the lifecycle of the ML application.

alessya@whylabs.ai

@zalessya

bit.ly/whylogs
Help build the open
standard for data
logging!

Contenu connexe

Tendances

Oct 2011 CHADNUG Presentation on Hadoop

Josh Patterson

The ever-increasing interest around deep learning and neural networks has led to a vast increase in processing frameworks like TensorFlow and PyTorch. These libraries are built around the idea of a computational graph that models the dataflow of individual units. Because tensors are their basic computational unit, these frameworks can run efficiently on hardware accelerators (e.g. GPUs).Traditional machine learning (ML) such as linear regressions and decision trees in scikit-learn cannot currently be run on GPUs, missing out on the potential accelerations that deep learning and neural networks enjoy. In this talk, we’ll show how you can use Hummingbird to achieve 1000x speedup in inferencing on GPUs by converting your traditional ML models to tensor-based models (PyTorch andTVM). https://github.com/microsoft/hummingbird This talk is for intermediate audiences that use traditional machine learning and want to speedup the time it takes to perform inference with these models. After watching the talk, the audience should be able to use ~5 lines of code to convert their traditional models to tensor-based models to be able to try them out on GPUs. Outline: Introduction of what ML inference is (and why it’s different than training) Motivation: Tensor-based DNN frameworks allow inference on GPU, but “traditional” ML frameworks do not Why “traditional” ML methods are important Introduction of what Hummingbirddoes and main benefits Deep dive on how traditional ML models are built Brief intro onhow Hummingbird converter works Example of how Hummingbird can convert a tree model into a tensor-based model Other models Demo Status Q&A

Tensors Are All You Need: Faster Inference with Hummingbird

Databricks

Join this session to hear from the Photon product and engineering team talk about the latest developments with the project. As organizations embrace data-driven decision-making, it has become imperative for them to invest in a platform that can quickly ingest and analyze massive amounts and types of data. With their data lakes, organizations can store all their data assets in cheap cloud object storage. But data lakes alone lack robust data management and governance capabilities. Fortunately, Delta Lake brings ACID transactions to your data lakes – making them more reliable while retaining the open access and low storage cost you are used to. Using Delta Lake as its foundation, the Databricks Lakehouse platform delivers a simplified and performant experience with first-class support for all your workloads, including SQL, data engineering, data science & machine learning. With a broad set of enhancements in data access and filtering, query optimization and scheduling, as well as query execution, the Lakehouse achieves state-of-the-art performance to meet the increasing demands of data applications. In this session, we will dive into Photon, a key component responsible for efficient query execution. Photon was first introduced at Spark and AI Summit 2020 and is written from the ground up in C++ to take advantage of modern hardware. It uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications — all natively on your data lake. Photon is fully compatible with the Apache Spark™ DataFrame and SQL APIs to ensure workloads run seamlessly without code changes. Come join us to learn more about how Photon can radically speed up your queries on Databricks.

Radical Speed for SQL Queries on Databricks: Photon Under the Hood

Databricks

AT&T has been involved in AI from the beginning, with many firsts; “first to coin the term AI”, “inventors of R”, “foundational work on Conv. Neural Nets”, etc. and we have applied AI to hundreds of solutions. Today we are modernizing these AI solutions in the cloud with the help of Databricks and a variety of in-house developments. This talk will highlight our AI modernization effort along with its application to Fraud which is one of our biggest benefitting applications.

AI Modernization at AT&T and the Application to Fraud with Databricks

Databricks

Streaming Inference with Apache Beam and TFX

Databricks

Number 2 in the Data Science for Dummies series - We'll predict Titanic survival with Databricks, python and MLSpark. These are the slides only (excuse the Powerpoint animation issues) - check out the actual tech talk on YouTube: https://rodneyjoyce.home.blog/2019/05/03/data-science-for-dummies-machine-learning-with-databricks-python-sparkml-tech-talk-1-of-7/) If you have not used Databricks before check out the first talk - Databricks for Dummies. Here's the rest of the series: https://rodneyjoyce.home.blog/tag/data-science-for-dummies/ 1) Data Science overview with Databricks 2) Titanic survival prediction with Azure Machine Learning Studio + Kaggle 3) Data Engineering with Titanic dataset + Databricks + Python 4) Titanic with Databricks + Spark ML 5) Titanic with Databricks + Azure Machine Learning Service 6) Titanic with Databricks + MLS + AutoML 7) Titanic with Databricks + MLFlow 8) Titanic with .NET Core + ML.NET 9) Deployment, DevOps/MLOps and Productionisation

Data Science for Dummies - Data Engineering with Titanic dataset + Databricks...

Rodney Joyce

Advanced SQL For Data Scientists

Databricks

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. We enable data scientists to deploy and operate their models independently, with minimal need for handoffs or gatekeeping. By writing a simple function and calling out to an intuitive API, data scientists can harness a suite of platform-provided tooling meant to make ML operations easy. In this talk, we will dive into the abstractions the Data Platform team has built to enable this. We will go over the interface data scientists use to specify a model and what that hooks into, including online deployment, batch execution on Spark, and metrics tracking and visualization.

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Databricks

Koalas is an open source project that provides pandas APIs on top of Apache Spark. Pandas is the standard tool for data science and it is typically the first step to explore and manipulate a data set, but pandas does not scale well to big data. Koalas fills the gap by providing pandas equivalent APIs that work on Apache Spark. There are also many libraries trying to scale pandas APIs, such as Vaex, Modin, and so on. Dask is one of them and very popular among pandas users, and also works on its own cluster similar to Koalas which is on top of Spark cluster. In this talk, we will introduce Koalas and its current status, and the comparison between Koalas and Dask, including benchmarking.

Koalas: How Well Does Koalas Work?

Databricks

A key benefit of serverless computing is that resources can be allocated on demand, but the quantity of resources to request, and allocate, for a job can profoundly impact its running time and cost. For a job that has not yet run, how can we provide users with an estimate of how the job’s performance changes with provisioned resources, so that users can make an informed choice upfront about cost-performance tradeoffs? This talk will describe several related research efforts at Microsoft to address this question. We focus on optimizing the amount of computational resources that control a data analytics query’s achieved intra-parallelism. These use machine learning models on query characteristics to predict the run time or Performance Characteristic Curve (PCC) as a function of the maximum parallelism that the query will be allowed to exploit. The AutoToken project uses models to predict the peak number of tokens (resource units) that is determined by the maximum parallelism that the recurring SCOPE job can ever exploit while running in Cosmos, an Exascale Big Data analytics platform at Microsoft. AutoToken_vNext, or TASQ, predicts the PCC as a function of the number of allocated tokens (limited parallelism). The AutoExecutor project uses models to predict the PCC for Apache Spark SQL queries as a function of the number of executors. The AutoDOP project uses models to predict the run time for SQL Server analytics queries, running on a single machine, as a function of their maximum allowed Degree Of Parallelism (DOP). We will present our approaches and prediction results for these scenarios, discuss some common challenges that we handled, and outline some open research questions in this space.

Predicting Optimal Parallelism for Data Analytics

Databricks

Equipment maintenance log of the global fleet is traditionally maintained using legacy infrastructure and data models, which limit the ability to extract insights at scale. However, to impact the bottom line, it is critical to ingest and enrich global fleet data to generate data driven guidance for operations. The impact of such insights is projected to be millions of dollars per annum. To this end, we leverage Databricks to perform machine learning at scale, including ingesting (structured and unstructured data) from legacy systems, and then sifting through millions of nonlinearly growing records to extract insights using NLP. The insights enable outlier identification, capacity planning, prioritization of cost reduction opportunities, and the discovery process for cross-functional teams.

NLP-Focused Applied ML at Scale for Global Fleet Analytics at ExxonMobil

Databricks

Zipline - A Declarative Feature Engineering Framework

Databricks

EFSA is the European agency providing independent scientific advice on existing and emerging risks across the entire food chain. On 27/03/2021 a new EU regulation (EU 2019/1381) has been enacted, requiring EFSA to significantly increase the transparency of its risk assessment processes towards all citizens. To comply with this new regulation, delaware has been supporting EFSA in undergoing a large Digital Transformation program. We have been designing and rolling-out a modern data platform running on Azure and powered by Databricks. This platform acts as a central control tower brokering data between a variety of applications. It is built around modularity principles, making it adaptable and versatile while keeping the overall ecosystem aligned w.r.t. changing processes and data models. At the heart of the platform lie two important patterns: 1. An Event Driven Architecture (EDA): enabling an extremely loosely coupled system landscape. By centrally brokering events near real-time, consumer applications can react immediately to events from producer applications as they occur. Event producers are decoupled from consumers via a publisher/subscribe mechanism. 2. A central data store built around a lakehouse architecture. The lakehouse collects, organizes and serves data across all stages of the data processing cycle, all data types and all data volumes. Events streams from the EDA layer feed into the store as curated data blocks and are complemented by other sources. This store in turn feeds into APIs, reporting and applications, including the new Open EFSA portal: a public website developed by delaware hosting all relevant scientific data, updated in near real-time. At delaware we are very excited about this project and proud of what we have achieved with EFSA so far.

Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA

Databricks

Tuning ML Models: Scaling, Workflows, and Architecture

Databricks

This talk describes migrating a large random forest classifier from scikit-learn to Spark's MLlib. We cut training time from 2 days to 2 hours, reduced failed runs, and track experiments better with MLflow. Kount provides certainty in digital interactions like online credit card transactions. One of our scores uses a random forest classifier with 250 trees and 100,000 nodes per tree. We used scikit-learn to train using 60 million samples that each contained over 150 features. The in-memory requirements exceeded 750 GB, took 2 days, and were not robust to disruption in our database or training execution. To migrate workflow to Spark, we built a 6-node cluster with HDFS. This provides 1.35 TB of RAM and 484 cores. Using MLlib and parallelization, the training time for our random forests are now less than 2 hours. Training data stays in our production environment, which used to require a deploy cycle to move locally-developed code onto our training server. The new implementation uses Jupyter notebooks for remote development with server-side execution. MLflow tracks all input parameters, code, and git revision number, while the performance and model itself are retained as experiment artifacts. The new workflow is robust to service disruption. Our training pipeline begins by pulling from a Vertica database. Originally, this single connection took over 8 hours to complete with any problem causing a restart. Using sqoop and multiple connections, we pull the data in 45 minutes. The old technique used volatile storage and required the data for each experiment. Now, we pull the data from Vertica one time and then reload much faster from HDFS. While a significant undertaking, moving to the Spark ecosystem converted an ad hoc and hands-on training process into a fully repeatable pipeline that meets regulatory and business goals for traceability and speed. Speaker: Josh Johnston

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

Databricks

For more than 6 years, Workday has been building various analytics products powered by Apache Spark. At the core of each product offering, customers use our UI to create data prep pipelines, which are then compiled to DataFrames and executed by Spark under the hood. As we built out our products, however, we started to notice places where vanilla Spark is not suitable for our workloads. For example, because our Spark plans are programmatically generated, they tend to be very complex, and often result in tens of thousands of operators. Another common issue is having case statements with thousands of branches, or worse, nested expressions containing such case statements. With the right combination of these traits, the final DataFrame can easily take Catalyst hours to compile and optimize – that is, if it doesn’t first cause the driver JVM to run out of memory. In this talk, we discuss how we addressed some of our pain points regarding complex pipelines. Topics covered include memory-efficient plan logging, using common subexpression elimination to remove redundant subplans, rewriting Spark’s constraint propagation mechanism to avoid exponential growth of filter constraints, as well as other performance enhancements made to Catalyst rules. We then apply these changes to several production pipelines, showcasing the reduction of time spent in Catalyst, and list out ideas for further improvements. Finally, we share tips on how you too can better handle complex Spark plans.

Optimizing the Catalyst Optimizer for Complex Plans

Databricks

"GOJEK, the Southeast Asian super-app, has seen an explosive growth in both users and data over the past three years. Today the technology startup uses big data powered machine learning to inform decision-making in its ride-hailing, lifestyle, logistics, food delivery, and payment products. From selecting the right driver to dispatch, to dynamically setting prices, to serving food recommendations, to forecasting real-world events. Hundreds of millions of orders per month, across 18 products, are all driven by machine learning. Building production grade machine learning systems at GOJEK wasn't always easy. Data processing and machine learning pipelines were brittle, long running, and had low reproducibility. Models and experiments were difficult to track, which led to downstream problems in production during serving and model evaluation. In this talk we will cover these and other challenges that we faced while trying to scale end-to-end machine learning systems at GOJEK. We will then introduce MLflow and explore the key features that make it useful as part of an ML platform. Finally, we will show how introducing MLflow into the ML life cycle has helped to solve many of the problems we faced while scaling machine learning at GOJEK. "

Scaling Ride-Hailing with Machine Learning on MLflow

Databricks

Splice Machine is an ANSI-SQL Relational Database Management System (RDBMS) on Apache Spark. It has proven low-latency transactional processing (OLTP) as well as analytical processing (OLAP) at petabyte scale. It uses Spark for all analytical computations and leverages HBase for persistence. This talk highlights a new Native Spark Datasource - which enables seamless data movement between Spark Data Frames and Splice Machine tables without serialization and deserialization. This Spark Datasource makes machine learning libraries such as MLlib native to the Splice RDBMS . Splice Machine has now integrated MLflow into its data platform, creating a flexible Data Science Workbench with an RDBMS at its core. The transactional capabilities of Splice Machine integrated with the plethora of DataFrame-compatible libraries and MLflow capabilities manages a complete, real-time workflow of data-to-insights-to-action. In this presentation we will demonstrate Splice Machine's Data Science Workbench and how it leverages Spark and MLflow to create powerful, full-cycle machine learning capabilities on an integrated platform, from transactional updates to data wrangling, experimentation, and deployment, and back again.

Splice Machine's use of Apache Spark and MLflow

Databricks

Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx

Lex Avstreikh

Building a machine learning model is an iterative process. A data scientist will build many tens to hundreds of models before arriving at one that meets some acceptance criteria. However, the current style of model building is ad-hoc and there is no practical way for a data scientist to manage models that are built over time. In addition, there are no means to run complex queries on models and related data. In this talk, we present ModelDB, a novel end-to-end system for managing machine learning (ML) models. Using client libraries, ModelDB automatically tracks and versions ML models in their native environments (e.g. spark.ml, scikit-learn). A common set of abstractions enable ModelDB to capture models and pipelines built across different languages and environments. The structured representation of models and metadata then provides a platform for users to issue complex queries across various modeling artifacts. Our rich web frontend provides a way to query ModelDB at varying levels of granularity. ModelDB has been open-sourced at https://github.com/mitdbg/modeldb.

ModelDB: A System to Manage Machine Learning Models: Spark Summit East talk b...

Spark Summit

Tendances (20)

Oct 2011 CHADNUG Presentation on Hadoop

Tensors Are All You Need: Faster Inference with Hummingbird

Radical Speed for SQL Queries on Databricks: Photon Under the Hood

AI Modernization at AT&T and the Application to Fraud with Databricks

Streaming Inference with Apache Beam and TFX

Data Science for Dummies - Data Engineering with Titanic dataset + Databricks...

Advanced SQL For Data Scientists

The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix

Koalas: How Well Does Koalas Work?

Predicting Optimal Parallelism for Data Analytics

NLP-Focused Applied ML at Scale for Global Fleet Analytics at ExxonMobil

Zipline - A Declarative Feature Engineering Framework

Building the Foundations of an Intelligent, Event-Driven Data Platform at EFSA

Tuning ML Models: Scaling, Workflows, and Architecture

Moving a Fraud-Fighting Random Forest from scikit-learn to Spark with MLlib, ...

Optimizing the Catalyst Optimizer for Complex Plans

Scaling Ride-Hailing with Machine Learning on MLflow

Splice Machine's use of Apache Spark and MLflow

Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx

ModelDB: A System to Manage Machine Learning Models: Spark Summit East talk b...

Similaire à The Critical Missing Component in the Production ML Stack

Strata + Hadoop 2015 Slides

Jun Liu

Talking Data is the largest independent big data service company in China. Their network covers 70% of the mobile services nationwide with 3 billion ad clicks per day. Amongst those clicks, 90% are potentially fraudulent. Click fraud is happening at an overwhelming volume leading to misusage of data and wasting money. Hence, Kaggle (a platform for predictive modeling and analytics competitions from the U.S.) has partnered up with TalkingData to help resolve this issue. This paper is to build predictive analysis models using traditional and Big Data methods to determine whether a smartphone app will be downloaded after clicking an advertisement. We have used data named “TalkingData AdTracking Fraud Detection Challenge”, which is of 7GB and given by a Kaggle competition. Four classification models are implemented with this massive data set in order to predict fraud in both traditional and Big Data methods. We define it fraud when the user clicked on an advertisement without downloading. The traditional platform has a resource limitation to build models with data set over a giga-byte so that we generate a sample data for the traditional models and adopt the full data set for the models in the Big Data Spark ML systems. We also present the accuracy and performance of the models implemented in both traditional and Big Data systems.

AdClickFraud_Bigdata-Apic-Ist-2019

Neha gupta

Database Shootout: What's best for BI?

Jos van Dongen

Purchasing a dedicated server to SQL Server is still a necessary operation. The cloud is a great choice but if you need to create a data warehouse of non-trivial size or if you have the need for optimal performance and control of your production database server, the choice of on-premise server is still an optimal choice. So, how not to throw away money on unnecessary hardware? In this session we will see how each component works together to form a balanced hardware (this is the key word!), without bottlenecks, maximizing the investment made. We'll talk about SAN, CPU, HBA, Fibre Channel, Memory and everything you thought you knew well...

Hardware planning & sizing for sql server

Davide Mauri

Sql server performance tuning and optimization

Manish Rawat

Red Hat Storage Day Atlanta - Designing Ceph Clusters Using Intel-Based Hardw...

Red_Hat_Storage

A data logger (also data logger or data recorder) is an electronic device that records data over time or in relation to location either with a built in instrument or sensor or via external instruments and sensors. Increasingly, but not entirely, they are based on a digital processor (or computer). They generally are small, battery powered, portable, and equipped with a microprocessor, internal memory for data storage, and sensors. Some data loggers interface with a personal computer and utilize software to activate the data logger and view and analyze the collected data, while others have a local interface device (keypad, LCD) and can be used as a stand-alone device. Data loggers vary between general purpose types for a range of measurement applications to very specific devices for measuring in one environment or application type only. It is common for general purpose types to be programmable; however, many remain as static machines with only a limited number or no changeable parameters. Electronic data loggers have replaced chart recorders in many applications.

8 Channel Analog Data Logger

Raghav Shetty

Entenda de onde vem toda a potência do Intel® Xeon Phi™

Intel Software Brasil

Soprex framework on .net in action

Milan Vukoje

An introduction to column store indexes and batch mode

Chris Adkin

Practical SQL query monitoring and optimization Today the project owners demand results as soon as possible and most often - for yesterday. Time to market is crucial and it is practical to deliver bit-by-bit, get feedback and grow with the number of your customers. But as the project grows, the team does too and not all have the same expertise. As well rarely in the beginning the requirements clear enough to allow performance-wise SQL interaction. In most cases there does not exist an ORM that can solve this task for you and you will need to have hard T-SQL writer in the team. If you already know this story or are going this way then in this practical session we will share how to monitor, measure and optimize your SQL code and DB layer interaction.

Practical SQL query monitoring and optimization

Ivo Andreev

Maximizing Database Tuning in SAP SQL Anywhere

SAP Technology

With SQL server 2014, Microsoft introduced In-Memory OLTP, codename Hekaton. This technology contains revolutionary concepts for optimizing the performance of OLTP workloads via natively compiled stored procedures,memory optimized tables and much more. This session will give you an overview of the different aspects of In-memory OLTP and how this is integrated in the SQL server engine. By giving demo's we will show you what is (im)possible with this new feature and how this really can increase your database performance!

SQL Track: In Memory OLTP in SQL Server

ITProceed

Do you know The Cloud Girl? She makes the cloud come alive with pictures and storytelling. The Cloud Girl, Priyanka Vergadia, Chief Content Officer @Google, joins us to tell us about Scaleable Data Analytics in Google Cloud. Maybe, with her explanation, we'll finally understand it! Priyanka is a technical storyteller and content creator who has created over 300 videos, articles, podcasts, courses and tutorials which help developers learn Google Cloud fundamentals, solve their business challenges and pass certifications! Checkout her content on Google Cloud Tech Youtube channel. Priyanka enjoys drawing and painting which she tries to bring to her advocacy. Check out her website The Cloud Girl: https://thecloudgirl.dev/ and her new book: https://www.amazon.com/Visualizing-Google-Cloud-Illustrated-References/dp/1119816327

GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...

James Anderson

Ml ops and the feature store with hopsworks, DC Data Science Meetup

Jim Dowling

A data warehouse is a database designed for query and analysis rather than for transaction processing. An appropriate design leads to scalable, balanced and flexible architecture that is capable to meet both present and long-term future needs. This session covers a comparison of the main data warehouse architectures together with best practices for the logical and physical design that support staging, load and querying.

Data Warehouse Design and Best Practices

Ivo Andreev

Oracle Query Optimizer - An Introduction

adryanbub

Practical businerss intelligence in share point 2013

Ivan Sanders

Devices from the IoT realm generate data in a rate and magnitude that make it practically impossible to retrieve valuable information without support of adequate AI engines. Storing and serving billions of data measurements over time is also a non-trivial task addressed by the special class of Time Series DBs. Out of these, InfluxDB has the largest popularity, provides comprehensive documentation and above all - is available open source. As well Microsoft have recently released Azure Time Series Insights - cloud offering of a TS DB with the usability promises from the Microsoft brand. This session is about managing and understanding IoT data.

Time Series Databases for IoT (On-premises and Azure)

Ivo Andreev

An In-Depth Look at SAP SQL Anywhere Performance Features

SAP Technology

Similaire à The Critical Missing Component in the Production ML Stack (20)

Strata + Hadoop 2015 Slides

AdClickFraud_Bigdata-Apic-Ist-2019

Database Shootout: What's best for BI?

Hardware planning & sizing for sql server

Sql server performance tuning and optimization

Red Hat Storage Day Atlanta - Designing Ceph Clusters Using Intel-Based Hardw...

8 Channel Analog Data Logger

Entenda de onde vem toda a potência do Intel® Xeon Phi™

Soprex framework on .net in action

An introduction to column store indexes and batch mode

Practical SQL query monitoring and optimization

Maximizing Database Tuning in SAP SQL Anywhere

SQL Track: In Memory OLTP in SQL Server

GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...

Ml ops and the feature store with hopsworks, DC Data Science Meetup

Data Warehouse Design and Best Practices

Oracle Query Optimizer - An Introduction

Practical businerss intelligence in share point 2013

Time Series Databases for IoT (On-premises and Azure)

An In-Depth Look at SAP SQL Anywhere Performance Features

Plus de Databricks

DW Migration Webinar-March 2022.pptx

Databricks

The world of data architecture began with applications. Next came data warehouses. Then text was organized into a data warehouse. Then one day the world discovered a whole new kind of data that was being generated by organizations. The world found that machines generated data that could be transformed into valuable insights. This was the origin of what is today called the data lakehouse. The evolution of data architecture continues today. Come listen to industry experts describe this transformation of ordinary data into a data architecture that is invaluable to business. Simply put, organizations that take data architecture seriously are going to be at the forefront of business tomorrow. This is an educational event. Several of the authors of the book Building the Data Lakehouse will be presenting at this symposium.

Data Lakehouse Symposium | Day 1 | Part 1

Databricks

Data Lakehouse Symposium | Day 1 | Part 2

Databricks

Data Lakehouse Symposium | Day 2

Databricks

Data Lakehouse Symposium | Day 4

Databricks

In this session, learn how to quickly supplement your on-premises Hadoop environment with a simple, open, and collaborative cloud architecture that enables you to generate greater value with scaled application of analytics and AI on all your data. You will also learn five critical steps for a successful migration to the Databricks Lakehouse Platform along with the resources available to help you begin to re-skill your data teams.

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Databricks

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Databricks

Data scientists face numerous challenges throughout the data science workflow that hinder productivity. As organizations continue to become more data-driven, a collaborative environment is more critical than ever — one that provides easier access and visibility into the data, reports and dashboards built against the data, reproducibility, and insights uncovered within the data.. Join us to hear how Databricks’ open and collaborative platform simplifies data science by enabling you to run all types of analytics workloads, from data preparation to exploratory analysis and predictive analytics, at scale — all on one unified platform.

Learn to Use Databricks for Data Science

Databricks

In this talk, I will dive into the stage level scheduling feature added to Apache Spark 3.1. Stage level scheduling extends upon Project Hydrogen by improving big data ETL and AI integration and also enables multiple other use cases. It is beneficial any time the user wants to change container resources between stages in a single Apache Spark application, whether those resources are CPU, Memory or GPUs. One of the most popular use cases is enabling end-to-end scalable Deep Learning and AI to efficiently use GPU resources. In this type of use case, users read from a distributed file system, do data manipulation and filtering to get the data into a format that the Deep Learning algorithm needs for training or inference and then sends the data into a Deep Learning algorithm. Using stage level scheduling combined with accelerator aware scheduling enables users to seamlessly go from ETL to Deep Learning running on the GPU by adjusting the container requirements for different stages in Spark within the same application. This makes writing these applications easier and can help with hardware utilization and costs. There are other ETL use cases where users want to change CPU and memory resources between stages, for instance there is data skew or perhaps the data size is much larger in certain stages of the application. In this talk, I will go over the feature details, cluster requirements, the API and use cases. I will demo how the stage level scheduling API can be used by Horovod to seamlessly go from data preparation to training using the Tensorflow Keras API using GPUs. The talk will also touch on other new Apache Spark 3.1 functionality, such as pluggable caching, which can be used to enable faster dataframe access when operating from GPUs.

Stage Level Scheduling Improving Big Data and AI Integration

Databricks

In this talk, I would like to introduce an open-source tool built by our team that simplifies the data conversion from Apache Spark to deep learning frameworks. Imagine you have a large dataset, say 20 GBs, and you want to use it to train a TensorFlow model. Before feeding the data to the model, you need to clean and preprocess your data using Spark. Now you have your dataset in a Spark DataFrame. When it comes to the training part, you may have the problem: How can I convert my Spark DataFrame to some format recognized by my TensorFlow model? The existing data conversion process can be tedious. For example, to convert an Apache Spark DataFrame to a TensorFlow Dataset file format, you need to either save the Apache Spark DataFrame on a distributed filesystem in parquet format and load the converted data with third-party tools such as Petastorm, or save it directly in TFRecord files with spark-tensorflow-connector and load it back using TFRecordDataset. Both approaches take more than 20 lines of code to manage the intermediate data files, rely on different parsing syntax, and require extra attention for handling vector columns in the Spark DataFrames. In short, all these engineering frictions greatly reduced the data scientists’ productivity. The Databricks Machine Learning team contributed a new Spark Dataset Converter API to Petastorm to simplify these tedious data conversion process steps. With the new API, it takes a few lines of code to convert a Spark DataFrame to a TensorFlow Dataset or a PyTorch DataLoader with default parameters. In the talk, I will use an example to show how to use the Spark Dataset Converter to train a Tensorflow model and how simple it is to go from single-node training to distributed training on Databricks.

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Databricks

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Scaling your Data Pipelines with Apache Spark on Kubernetes

Databricks

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Databricks

We want to present multiple anti patterns utilizing Redis in unconventional ways to get the maximum out of Apache Spark.All examples presented are tried and tested in production at Scale at Adobe. The most common integration is spark-redis which interfaces with Redis as a Dataframe backing Store or as an upstream for Structured Streaming. We deviate from the common use cases to explore where Redis can plug gaps while scaling out high throughput applications in Spark. Niche 1 : Long Running Spark Batch Job – Dispatch New Jobs by polling a Redis Queue · Why? o Custom queries on top a table; We load the data once and query N times · Why not Structured Streaming · Working Solution using Redis Niche 2 : Distributed Counters · Problems with Spark Accumulators · Utilize Redis Hashes as distributed counters · Precautions for retries and speculative execution · Pipelining to improve performance

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Databricks

Machine learning (ML) models are typically part of prediction queries that consist of a data processing part (e.g., for joining, filtering, cleaning, featurization) and an ML part invoking one or more trained models. In this presentation, we identify significant and unexplored opportunities for optimization. To the best of our knowledge, this is the first effort to look at prediction queries holistically, optimizing across both the ML and SQL components. We will present Raven, an end-to-end optimizer for prediction queries. Raven relies on a unified intermediate representation that captures both data processing and ML operators in a single graph structure. This allows us to introduce optimization rules that (i) reduce unnecessary computations by passing information between the data processing and ML operators (ii) leverage operator transformations (e.g., turning a decision tree to a SQL expression or an equivalent neural network) to map operators to the right execution engine, and (iii) integrate compiler techniques to take advantage of the most efficient hardware backend (e.g., CPU, GPU) for each operator. We have implemented Raven as an extension to Spark’s Catalyst optimizer to enable the optimization of SparkSQL prediction queries. Our implementation also allows the optimization of prediction queries in SQL Server. As we will show, Raven is capable of improving prediction query performance on Apache Spark and SQL Server by up to 13.1x and 330x, respectively. For complex models, where GPU acceleration is beneficial, Raven provides up to 8x speedup compared to state-of-the-art systems. As part of the presentation, we will also give a demo showcasing Raven in action.

Raven: End-to-end Optimization of ML Prediction Queries

Databricks

Semantic segmentation is the classification of every pixel in an image/video. The segmentation partitions a digital image into multiple objects to simplify/change the representation of the image into something that is more meaningful and easier to analyze [1][2]. The technique has a wide variety of applications ranging from perception in autonomous driving scenarios to cancer cell segmentation for medical diagnosis. Exponential growth in the datasets that require such segmentation is driven by improvements in the accuracy and quality of the sensors generating the data extending to 3D point cloud data. This growth is further compounded by exponential advances in cloud technologies enabling the storage and compute available for such applications. The need for semantically segmented datasets is a key requirement to improve the accuracy of inference engines that are built upon them. Streamlining the accuracy and efficiency of these systems directly affects the value of the business outcome for organizations that are developing such functionalities as a part of their AI strategy. This presentation details workflows for labeling, preprocessing, modeling, and evaluating performance/accuracy. Scientists and engineers leverage domain-specific features/tools that support the entire workflow from labeling the ground truth, handling data from a wide variety of sources/formats, developing models and finally deploying these models. Users can scale their deployments optimally on GPU-based cloud infrastructure to build accelerated training and inference pipelines while working with big datasets. These environments are optimized for engineers to develop such functionality with ease and then scale against large datasets with Spark-based clusters on the cloud.

Processing Large Datasets for ADAS Applications using Apache Spark

Databricks

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences. What are we storing? Multi Source – Multi Channel Problem Data Representation and Nested Schema Evolution Performance Trade Offs with Various formats Go over anti-patterns used (String FTW) Data Manipulation using UDFs Writer Worries and How to Wipe them Away Staging Tables FTW Datalake Replication Lag Tracking Performance Time!

Massive Data Processing in Adobe Using Delta Lake

Databricks

Detecting advanced email attacks at scale is a challenging ML problem, particularly due to the rarity of attacks, adversarial nature of the problem, and scale of data. In order to move quickly and adapt to the newest threat we needed to build a Continuous Integration / Continuous Delivery pipeline for the entire ML detection stack. Our goal is to enable detection engineers and data scientists to make changes to any part of the stack including joined datasets for hydration, feature extraction code, detection logic, and develop/train ML models. In this talk, we discuss why we decided to build this pipeline, how it is used to accelerate development and ensure quality, and dive into the nitty-gritty details of building such a system on top of an Apache Spark + Databricks stack.

Machine Learning CI/CD for Email Attack Detection

Databricks

Sarah: CEO-Finance-Report pipeline seems to be slow today. Why Jeeves: SparkSQL query dbt_fin_model in CEO-Finance-Report is running 53% slower on 2/28/2021. Data skew issue detected. Issue has not been seen in last 90 days. Jeeves: Adding 5 more nodes to cluster recommended for CEO-Finance-Report to finish in its 99th percentile time of 5.2 hours. Who is Jeeves? An experienced Spark developer? A seasoned administrator? No, Jeeves is a chatbot created to simplify data operations management for enterprise Spark clusters. This chatbot is powered by advanced AI algorithms and an intuitive conversational interface that together provide answers to get users in and out of problems quickly. Instead of being stuck to screens displaying logs and metrics, users can now have a more refreshing experience via a two-way conversation with their own personal Spark expert. We presented Jeeves at Spark Summit 2019. In the two years since, Jeeves has grown up a lot. Jeeves can now learn continuously as telemetry information streams in from more and more applications, especially SQL queries. Jeeves now “knows” about data pipelines that have many components. Jeeves can also answer questions about data quality in addition to performance, cost, failures, and SLAs. For example: Tom: I am not seeing any data for today in my Campaign Metrics Dashboard. Jeeves: 3/5 validations failed on the cmp_kpis table on 2/28/2021. Run of pipeline cmp_incremental_daily failed on 2/28/2021. This talk will give an overview of the newer capabilities of the chatbot, and how it now fits in a modern data stack with the emergence of new data roles like analytics engineers and machine learning engineers. You will learn how to build chatbots that tackle your complex data operations challenges.

Jeeves Grows Up: An AI Chatbot for Performance and Quality

Databricks

Hyperparameter tuning is critical in model development. And its general form: parameter tuning with an objective function is also widely used in industry. On the other hand, Apache Spark can handle massive parallelism, and Apache Spark ML is a solid machine learning solution. But we have not seen a general and intuitive distributed parameter tuning solution based on Apache Spark, why? Not every tuning problem is on Apache Spark ML models. How can Apache Spark handle general models? Not every tuning problem is a parallelizable grid or random search. Bayesian optimization is sequential, how can Apache Spark help in this case? Not every tuning problem is single epoch, deep learning is not. How to fit algos such as hyperband and ASHA into Apache Spark? Not every tuning problem is a machine learning problem, for example simulation + tuning is also common. How to generalize? In this talk, we are going to show how using Fugue-Tune and Apache Spark together can eliminate these painpoints Fugue-Tune like Fugue, is a “super framework” – an absraction layer unifying existing solutions such as Hyperopt and Optuna It firstly models the general tuning problems, independent from machine learning It is designed for both small and large scale problems. It can always fully parallelize the distributable part of a tuning problem It works for both classical and deep learning models. With Fugue, running hyperband and ASHA becomes possible on Apache Spark. In the demo, you will see how to do any type of tuning in a consistent, intuitive, scalable and minimal way. And you will see a live demo of the amazing performance.

Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue

Databricks

This presentation will explore the new work in Spark 3.1 adding the concept of graceful decommissioning and how we can use this to improve Spark’s performance in both dynamic allocation and spot/preemptable instances. Together we’ll explore how Spark’s dynamic allocation has evolved over time, and why the different changes have been needed. We’ll also look at the multi-company collaboration that resulted in being able to deliver this feature and I’ll end with encouraging pointers on how to get more involved in Spark’s development.

Improving Apache Spark for Dynamic Allocation and Spot Instances

Databricks

Plus de Databricks (20)

DW Migration Webinar-March 2022.pptx

Data Lakehouse Symposium | Day 1 | Part 1

Data Lakehouse Symposium | Day 1 | Part 2

Data Lakehouse Symposium | Day 2

Data Lakehouse Symposium | Day 4

5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop

Democratizing Data Quality Through a Centralized Platform

Learn to Use Databricks for Data Science

Stage Level Scheduling Improving Big Data and AI Integration

Simplify Data Conversion from Spark to TensorFlow and PyTorch

Scaling your Data Pipelines with Apache Spark on Kubernetes

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink

Raven: End-to-end Optimization of ML Prediction Queries

Processing Large Datasets for ADAS Applications using Apache Spark

Massive Data Processing in Adobe Using Delta Lake

Machine Learning CI/CD for Email Attack Detection

Jeeves Grows Up: An AI Chatbot for Performance and Quality

Intuitive & Scalable Hyperparameter Tuning with Apache Spark + Fugue

Improving Apache Spark for Dynamic Allocation and Spot Instances

Dernier

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

nirzagarg

Yilin Xia (yilinx2@illinois.edu), Shawn Bowers (bowers@gonzaga.edu), Lan Li (lanl2@illinois.edu), and Bertram Ludäscher (ludaesch@illinois.edu) Presented at IDCC-2024 in Edinburg. ABSTRACT. We propose a new approach for modeling and reconciling conflicting data cleaning actions. Such conflicts arise naturally in collaborative data curation settings where multiple experts work independently and then aim to put their efforts together to improve and accelerate data cleaning. The key idea of our approach is to model conflicting updates as a formal argumentation framework (AF). Such argumentation frameworks can be automatically analyzed and solved by translating them to a logic program PAF whose declarative semantics yield a transparent solution with many desirable properties, e.g., uncontroversial updates are accepted, unjustified ones are rejected, and the remaining ambiguities are exposed and presented to users for further analysis. After motivating the problem, we introduce our approach and illustrate it with a detailed running example introducing both well-founded and stable semantics to help understand the AF solutions. We have begun to develop open source tools and Jupyter notebooks that demonstrate the practicality of our approach. In future work we plan to develop a toolkit for conflict resolution that can be used in conjunction with OpenRefine, a popular interactive data cleaning tool.

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...

Bertram Ludäscher

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Service Booking Now open +91- 9332606886 Pune VIP Call Girls is highly qualified professionals. #S10 They will offer only top quality services and never try to scam or drain you of money; in fact, they strive to give complete satisfaction so that each cent spent was worth its worth. Furthermore, these professionals pride themselves in keeping customer details confidential; any information gleaned will never be shared with any third party.$V15 Two shots with one girl: ₹4500/in-call, ₹7500/out-call Body to body massage with : ₹5500/in-call Full night for one person: ₹7000/in-call, ₹12000/out-call •𝐂𝐨𝐥𝐥𝐞𝐠𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐇𝐨𝐮𝐬𝐞𝐰𝐢𝐯𝐞𝐬 •𝐇𝐢𝐠𝐡 𝐏𝐫𝐨𝐟𝐢𝐥𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐑𝐚𝐦𝐩 𝐌𝐨𝐝𝐞𝐥𝐬 •𝐅𝐨𝐫𝐞𝐢𝐠𝐧𝐞𝐫 𝐆𝐢𝐫𝐥𝐬 𝐎𝐮𝐫 𝐞𝐬𝐜𝐨𝐫𝐭 𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 •𝐇𝐉 (𝐇𝐚𝐧𝐝 𝐉𝐨𝐛) •𝐁𝐉 (𝐁𝐥𝐨𝐰𝐣𝐨𝐛) # Completion # (Oral To Completion) # Special Massage # O-Level (Oral ) # Oral With A Noncondom) # COB (Come On Body) # Extraball (Have Many Times) # All Meetings 100%Safe

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

kumargunjan9515

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

gajnagarg

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

amy56318795

Klinik_ Apotek Onlin 085657271886 Solusi Menggugurkan Masalah Kehamilan Anda Jual Obat Aborsi Asli KLINIK ABORSI TERPEECAYA _ Jual Obat Aborsi Cytotec Misoprostol Asli 100% Ampuh Hanya 3 Jam Langsung Gugur || OBAT PENGGUGUR KANDUNGAN AMPUH MANJUR OBAT ABORSI OLINE" APOTIK Jual Obat Cytotec, Gastrul, Gynecoside Asli Ampuh. JUAL ” Obat Aborsi Tuntas | Obat Aborsi Manjur | Obat Aborsi Ampuh | Obat Penggugur Janin | Obat Pencegah Kehamilan | Obat Pelancar Haid | Obat terlambat Bulan | Ciri Obat Aborsi Asli | Obat Telat Bulan | Pil Aborsi Asli | Cara Menggugurkan Konten | Cara Aborsi Tuntas | Harga Obat Aborsi Asli | Pil Aborsi | Jual Obat Aborsi Cytotec | Cara Aborsi Sendiri | Cara Aborsi Usia 1 Bulan | Cara Aborsi Usia 2 Tahun | Cara Aborsi Usia 3 Bulan | Obat Aborsi Usia 4 Bulan | Cara Abrasi Usia 5 Bulan | Cara Menggugurkan Konten | Kandungan Obat Penggugur | Cara Menghitung Usia Konten | Cara Mengatasi Terlambat Bulan | Penjual Obat Aborsi Asli | Obat Aborsi Garansi | Kandungan Obat Peluntur | Obat Telat Datang Bulan | Obat Telat Haid | Obat Aborsi Paling Murah | Klinik Jual Obat Aborsi | Jual Pil Cytotec | Apotik Jual Obat Aborsi | Kandungan Dokter Abrasi | Cara Aborsi Cepat | Jual Obat Aborsi Bergaransi | Jual Obat Cytotec Asli | Obat Aborsi Aman Manjur | Obat Misoprostol Cytotec Asli. "APA ITU ABORSI" “Aborsi Adalah dengan membendung hormon yang di perlukan untuk mempertahankan kehamilan yaitu hormon progesteron, karena hormon ini dibendung, maka jalur kehamilan mulai membuka dan leher rahim menjadi melunak,sehingga mengeluarkan darah yang merupakan tanda bahwa obat telah bekerja || maksimal 1 jam obat diminum || PENJELASAN OBAT ABORSI USIA 1 _7 BULAN Pada usia kandungan ini, pasien akan merasakan sakit yang sedikit tidak berlebihan || sekitar 1 jam ||. namun hanya akan terjadi pada saatdarah keluar merupakan pertanda menstruasi. Hal ini dikarenakan pada usiakandungan 3 bulan,janin sudah terbentuk sebesar kepalan tangan orang dewasa. Cara kerja obat aborsi : JUAL OBAT ABORSI AMPUH dosis 3 bulan secara umum sama dengan cara kerja || DOSIS OBAT ABORSI 2 bulan”, hanya berbedanya selain mengisolasijanin juga menghancurkan janin dengan formula methotrexate dikandungdidalamnya. Formula methotrexate ini sangat ampuh untuk menghancurkan janinmenjadi serpihan-serpihan kecil akan sangat berguna pada saat dikeluarkan nanti. APA ALASAN WANITA MELAKUKAN ABORSI? Aborsi di lakukan wanita hamil baik yang sudah menikah maupun belum menikah dengan berbagai alasan , akan tetapi alasan yang utama adalah alasan-alasan non medis (termasuk aborsi sendiri / di sengaja/ buatan] MELAYANI PEMESANAN OBAT ABORSI SETIAP HARI, SIAP KIRIM KESELURUH KOTA BESAR DI INDONESIA DAN LUAR NEGERI. HUBUNGI PEMESANAN LEBIH NYAMAN VIA WA/: 085657271886

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

ZurliaSoop

Lecture_2_Deep_Learning_Overview-newone1

ranjankumarbehera14

Statistics notes ,it includes mean to index numbers

suginr1

Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...

nirzagarg

+97470301568 Qatar THC Oil and weed in Qatar? Where can I get THC vape in Doha Qatar?WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in France,Germany, Poland Serbia,Romania, Ukraine WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia Qatar Where can I get weed in Qatar? Where can I get THC vape in Doha Qatar?WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in France,Germany, Poland Serbia,Romania, UkraineWhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia Qatar WhatsApp+97470301568 WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Qatar Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia #Singapore #Jordan #Ireland, #Belgium, #United Kingdom, #Iceland, #*Portugal, Spain, China, Japan, Turkey, Canada United States, Morocco, France,Germany, Poland Serbia,Romania, Ukraine, and all countries United Arab Emirates . Our team has succesfully delivered in 26 different countries . All marijuana and Cocaine is double vacuum packed before shipping, making it completely odorless to ensure that it arrives safely to your door. Our distribution crew is expert at making packages that blend in with the rest of the mail. We have also put into place many other security measures to ensure the security of our customers. buy weed Dubai +97470301568buy Weed Qatar #Buy Weed Kuwait #Buy Weed Bahrain #Buy #Weed #Oman #Buy Weed UAE #Buy Weed Abu Dhabi @Buy Weed Doha Qatar #@Buy Weed Ajman #@Buy Weed Online #@Buy Weed UK #@Buy Weed Iceland #*@Buy Weed All Countries Below are the various strains of kush available ; buy hash and Weed in dubai,abu dhabi,sharjah where to buy weed in doha,where can i find weed in jeddah,Can I get weed delivered to Riyadh?,Buy weed Online Jeddah Saudi Arabia,Buy Weed and THC Cannabis Oil online ,QATAR , DOHA buy kush in DOHA , buy kush in DOHAWeed in QATAR # DOHA Buy Weed and THC Cannabis Oil online who delivers at your own location in Qatar Doha ,Kuwait ,Dubai including cannabis / weed,Where can I find weed in Dubai as a tourist?,Is marijuana allowed in Dubai ? How much is medical marijuana in Dubai ?Is weed legal in Dubai ?How to get marijuana in Saudi Arabia Do people in Saudi Arabia smoke weed ? Is Hash legal in Saudi Arabia ? Where is marijuana the most illegal?Can you get weed in Baku? Dubai, United Arab Emirates Canabis smokers - Dubai, Buy Marijuana Products Online in UAE Desertcart ships the Marijuana products in Dubai ,Abu Dhabi, Sharjah, Al Ain, Ajman and more cities in UAE. Get unlimited free shipping in 164+ countries Buy Weed Products Online in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Buy Marijuana Saudi Arabia weed in jeddahis cbd legal in saudi arabia cali tins smoke in saudi arabia drug use saudi arabia Buy weed marijuana. White Widow OG #Kush Sensi Star x ak 47 Afghan Ku

+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...

Health

Abortion pills in Kuwait city !🌆+918761049707^) Where to get cytotec pills in salmiyah, WhatsApp '''(+918761049707)''' Explore Discreet and Safe Abortion Pill Options in Dubai, Empowering Women With Confidential and Medically Supervised Choices For Reproductive Health" Abortion Pills In Dubai, Abu Dhabi/Alain/Sharjah/RAK City Satwa-Mifepristone and Misoprostol Available in Dubai/Abu Dhabi - i Pills in Dubai and Cost Of Cytotec in WhatsApp (+918761049707) Abortion Pills in Dubai/Abu Dubai/ Abu Dhabi. Are you stranded with unwanted pregnancy in Dubai, Abu Dhabi , the United Arab Emirates (U.A.E), Qatar, Oman, Saudi Arabia or Kuwait? you can now contact us now on Whatsapp Dr AJ:+918761049707to buy safe abortion pills In Dubai, Abu Dhabi, Sharjah, Al Ain, Ajman, RAK City, Ras Al Khaimah and Fujairah to terminate an unwanted pregnancy in Dubai and the United Arab Emirates. Get your Discreet 100% Safe (+918761049707 )*Effective Abortion Pills For Sale in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE. BUY Mifepristone and Misoprostol (Cytotec), Mtp Kit In UAE. Abortion pills available in UAE (United Arab Emirates), Saudi Arabia, Kuwait, Oman, Bahrain and Qatar. Contact us today. +918761049707 -The UAE’s leading abortion care service in Dubai. Abortion Treatment. Medical Abortion. Surgical Abortion. Find A Clinic like Dr Maria Abortion clinic in Dubai We have Abortion Pills / Cytotec Tablets Available in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE., buy cytotec in Dubai abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain We sell original abortion medicine which includes: Cytotec 200mcg (Misoprostol), Mifepristone, Mifegest-kit, Misoclear, Emergency contraceptive pills, Morning after sex pills, ipills, pills to prevent pregnancy 72 hours after sex. All our pills are manufactured by reputable medical manufacturing companies like PFIZER. Medical abortion is easy and effective for everyone to perform in their own privacy. There are very few complications that may arise from medical abortion if one follows the right guidelines as instructed by the obstetrician. Abortion Pills in Dubai Can Now Be Offered at Dr Maria Abortion clinic in Dubai, F.D.A. Mifepristone and Misoprostol, the first of two drugs in medication abortions, previously had to be dispensed only by clinics, doctors or a few mail-order pharmacies like Dr Maria Abortion clinic in Dubai . Now, We can provide it. For the first time, retail pharmacies, like Dr Maria Abortion clinic in Dubai, will be Able to offer abortion pills in Dubai under a regulatory change made Tuesday by the Food and Drug Administration. The action could significantly expand access to abortion through medication. Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

ahmedjiabur940

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C 9548273370 Escort Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

HyderabadDolls

Building Real-Time Pipelines With FLaNK Timothy Spann, Principal Developer Advocate, Streaming - Cloudera Future of Data meetup, startup grind, AI Camp The combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines is extremely powerful, as demonstrated by this case study using the FLaNK-MTA project. The project leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA). FLaNK-MTA demonstrates how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making. Apache NiFi Apache Kafka Apache Flink Apache Iceberg LLM Generative AI Slack Postgresql

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Timothy Spann

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & Dating Escorts Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

HyderabadDolls

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort Service Available 24/7 Hire CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort S...

HyderabadDolls

#Dubai Call Girls Agency +971525547819 #Indian And Pakistani Call Girls Dubai #Dubai Indian Call Girls Agency Class Call Girls In Dubai #First Class Call Girls In Dubai #Full Massage Services Call Girls In Dubai #Al Jaddaf,Al Jaffiliya,Business Bay,Al Karama,Bur Dubai,Deira,Dubai,Palm Jumeirah,Al Wasl,Trade Centre,Dubai Mall,JBR,JVC,JLT,Discovery Garden #Dubai Call Girls Services Provide In Ajman_Dubai_RAK_UMQ_Fujairah_Abu_Dhabi#Indian #Tamil #Kerala #Russian #Philippine #Morocco #Thailand #English Models In Dubai #If You Want Serv#Dubai Pakistani Call Girls Agency #Beautiful Call Girls in Dubai #High ices Just Send Me Text On Whatsapp +971525547819 #Website Link http://Dubaicallgirls.pro https://chatwith.io/s/65d1df48b2992

Dubai Call Girls Peeing O525547819 Call Girls Dubai

kojalkojal131

原版定制【微信:153539019】《英国诺森比亚大学毕业证（NU毕业证书）》【微信:153539019】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。【业务选择办理准则】一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信153539019】文凭即可二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信153539019】即可三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才【关于价格问题（保证一手价格）】我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子我给客户的都是第一手的代理价格，因为我想坦诚对待大家不想跟大家在价格方面浪费时间对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。

如何办理英国诺森比亚大学毕业证（NU毕业证书）成绩单原件一模一样

wsppdmt

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now Booking Contact Details :- WhatsApp Chat :- +91-7737669865 We offer all types of girls of your choice with space. Our escorts are fully cooperative and understand your needs. All types of call girls like Housewives, College girls,#K09 Russian girls, Muslim girls, Afghani girls, Bengali girls, Working girls, south Indian girls, Punjabi girls, etc. In-Call: — You Can Reach At Our Place in Bangalore Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: — Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Hygienic: — Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Bangalore Our Services and Rates: – One Shot — 2500/in call (time ½ hour), 5000/out call Two shot with one girl — 5000/in call (time 1 hour), 6000/out call Body to body massage with sex- 3000/in call (time 1 hour) full night for one person– 8000/in call, 10000/out call (shot limit 4 shot) We are available 24*7 all days of the year

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now

gargpaaro

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We...

gajnagarg

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort Service Available 24/7 Hire Booking Contact Details :- WhatsApp Chat :- +91-6378878445 We offer all types of girls of your choice with space. Our escorts are fully cooperative and understand your needs.#K09 All types of call girls like Housewives, College girls,#K09 Russian girls, Muslim girls, Afghani girls, Bengali girls, Working girls, south Indian girls, Punjabi girls, etc. In-Call: — You Can Reach At Our Place in Bangalore Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: — Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Hygienic: — Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Bangalore Our Services and Rates: – One Shot — 2500/in call (time ½ hour), 5000/out call Two shot with one girl — 5000/in call (time 1 hour), 6000/out call Body to body massage with sex- 3000/in call (time 1 hour) full night for one person– 8000/in call, 10000/out call (shot limit 4 shot) We are available 24*7 all days of the year

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

gragchanchal546

Dernier (20)

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

Jual Obat Aborsi Surabaya ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

Lecture_2_Deep_Learning_Overview-newone1

Statistics notes ,it includes mean to index numbers

Top profile Call Girls In Tumkur [ 7014168258 ] Call Me For Genuine Models We...

+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort S...

Dubai Call Girls Peeing O525547819 Call Girls Dubai

如何办理英国诺森比亚大学毕业证（NU毕业证书）成绩单原件一模一样

Vadodara 💋 Call Girl 7737669865 Call Girls in Vadodara Escort service book now

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We...

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

The Critical Missing Component in the Production ML Stack

1. The critical missing component in the production ML stack Alessya Visnjic CEO, WhyLabs.ai

2. Agenda: ▪ ML Stack: what is missing? ▪ How to design data logging ▪ whylogs: open standard for data logging ▪ Use cases ▪ Q&A

3. ML models are full of surprises… … every surprise will launch a debugging expedition!

4. ML models routinely struggle in the wild…

5. ML Stack: moving massive volumes of data

6. Test Monitor Debug Document How do you data?

7. ML Stack is missing data & metadata logging Log metadata & statistical properties of data

8. Good data log should capture: Metadata Counts Statistics Distributions Stratified samples

9. Key properties of a data log: Lightweight Portable Mergeable Configurable Close to code

10. whylogs: logging for the ML stack bit.ly/whylogs

11. whylogs: a standard format for representing a snapshot of data bit.ly/whylogs

12. Profile and log a dataframe bit.ly/whylogs

13. Profile, log, and track with bit.ly/whylogs

14. Feature name count max min stddev nunique null_count quantile_0.0000 … quantile_1.0000 chlorides 1199.0 0.611 0.012 0.044 134.0 0.0 0.012 … 0.611 quality 1199 8.000 3.000 0.785 6.0 0.0 3.000 … 8.000 alcohol 1199 14.900 8.400 1.060 65.0 0.0 8.400 … 14.900 density 1199 1.004 0.997 0.001 390.0 0.0 0.990 … 1.004 pH 1199 4.010 2.890 0.153 82.0 0.0 2.890 … 4.010 Log rich statistics for each feature Each data log captures summary statistics, counters, distributions, metadata and custom metrics Sample of a flattened data log captured by whylogs on the Wine Quality dataset

15. Track data statistics across batches Distribution plot for one of the columns in the model input, collected at inference time Distribution of “free sulfur dioxide” feature over 20 inference batches of the Wine Quality model

16. Dataset Size # of entries # of features Memory consumption Output size Lending Club 1.6G 2.2M 151 14MB 7.4MB NYC Tickets 1.9G 10.8 43 14MB 2.3MB Pain pills 75GB 178M 42 15MB 2MB Run data logging without overhead Using streaming algorithms to capture data statistics, whylogs ensures a constant memory footprint, scales with the number of features in the dataframe, and outputs lightweight log files (json, protobuf, etc). Sample of whylogs benchmarks on public datasets

17. Whylogs profiles 100% of the data to accurately capture distributions. Calculating distributions from randomly sampled data is significantly less accurate. The chart presents median error for distributions estimated with whylogs vs. random sampling techniques. 0 0.1 0.2 0.2 0.3 0.4 Normal Normal discrete Normal outlier Uniform discrete Uniform Pareto Profiling Sampling Capture accurate data distributions

18. Test Monitor Debug Document You can data with whylogs!

19. Whylogs captures mergeable histograms for each feature. To catch distribution drift, continuously compare training distribution of a feature to the serving distribution. 0 200 400 600 800 1000 Training Serving Use case: training-serving distribution drift

20. Logging enables all key MLOps activities Once data is logged systematically, whylogs outputs can be used to test, monitor, and debug data. Use whylogs at any point of the ML stack and through the lifecycle of the ML application.

21. alessya@whylabs.ai @zalessya bit.ly/whylogs Help build the open standard for data logging!

The Critical Missing Component in the Production ML Stack

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à The Critical Missing Component in the Production ML Stack

Similaire à The Critical Missing Component in the Production ML Stack (20)

Plus de Databricks

Plus de Databricks (20)

Dernier

Dernier (20)

The Critical Missing Component in the Production ML Stack