Uber's data science workbench

•

9 likes•3,232 views

Ran Wei

https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/56711

Engineering

Uber’s Data Science Workbench
Randy Wei Peng Du

Mission
Unleash the productivity of the Data
Science community at Uber by
providing scalable infrastructure,
tools, customization and support.

Tools of the Trade: Jupyter Notebooks
Alternative to traditional CLIs
Interactive tool which combines
Prose (HTML Markdown),
Code (Py, R, Scala)
Visualization (charts, maps, tables)
Shareable artifact of knowledge
Hosted webapp
Notebook, Notes, Cells
Each cell is an executable line of code
Used for
Data exploration, Cleansing, Modeling
Dashboarding/reporting
HTML
Code
Output

Tools of the Trade: RStudio Server
Browser interface to a remote R server
Centrally manage compute infrastructure
IDE for R
Syntax highlight, code completion
Debugging
Charts
File Browser
RStudio also has Notebook functionality
R has a huge library repository
Used mostly for rapid prototyping of models
on small datasets (UbeR)
Data
Code
Output

Tools of the Trade: Apache Spark
Distributed statistical computing framework
Run R code without translating it to Java
Choice of Intelligent Decision, Insurance, etc
teams
Distributed machine learning framework
Easy to integrate with scientific Python
libraries
Choice of Fraud Detection, Sensing and
Perception, etc teams
SparkR PySpark

● Productivity
● Py, R, Scala interpreters in Jupyter
● Hosted RStudio support
● Version Control
● Custom libraries/environment
● Single-pane lifecycle mgmnt.
● PySpark, SparkR
Scale
● Scalable Jupyter Server infra.
● Large dist. computation backend
● Multitenancy
● File Persistence
● Security
Requirements
Ecosystem Integration
● Scheduling: Piper
● Dashboards: Shiny
● Data Exploration: Query engine API
● Deploy: Machine learning platform
● Chargeback: Monitoring platform
● Knowledge
● Search
● Access Controls
● Sharing Controls
● Publish
● Comments & Discussion
Scale Productivity
Social Ecosystem

State of the Union
Problem
● Data Scientists (DSs) start
at Uber with diverse
skillsets and backgrounds
● Precious time wasted in
infra. setup, version control,
search, sharing...
● Teams are building their
own solutions
Vision
● Web-based hub for all Data
Scientists at Uber
● Ability to centrally:
○ provision tools
○ leverage dist.
Backend
○ search, comment,
share
○ monitor
● Integrated with Uber’s data
ecosystem
● Dedicated SRE
Opportunity
● Find and reuse knowledge
● Opportunity for a dedicated
team to advocate for and
build tools needs to make
DSs hyper-productive
● Cloud experience
● Chargeback

Management Service
Create, Delete, Search, Share, Publish, Schedule
RStudio
(Docker)
Uber Mesos Infra Shared File System
MLlib
Worker
MLlib
Worker
MLlib
Worker
MLlib
Worker
MLlib
Worker
PySpark
Worker
MLlib
Worker
MLlib
Worker
SparkR
Worker
Uber spark
debugging
toolkit
Uber spark
development
toolkit
RStudio
(Docker)
RStudio
(Docker) RStudio
(Docker)
RStudio
(Docker)
Jupyter
(Docker)
Manage
Mesos
Spark
Architecture

Architecture
NB1
Application
Management
Service
session / file
management,
proxy
Mesos Cluster
Docker Container Hadoop
Cluster
(Hive, Presto,
Spark)
Distributed
ProcessingDocker Container
Docker Container
RStudio
Server
RStudio
Jupyter
Docker Container
NB1Jupyter
Server NB2
Web GUI

Data Science
Workbench
Uber ML platform Palette
Hive Cassandra
Spark
Spark SDK, Spark Debug
tool, Spark templates
Uber Ecosystem
Models
HDFS
Query
Runner
Production
PySpark
for ML
Data Visualization

What's hot

An informational, or statistical, constraint is a constraint such as a unique, primary key, foreign key, or check constraint that can be used by Apache Spark to improve query performance. Informational constraints are not enforced by the Spark SQL engine; rather, they are used by Catalyst to optimize the query processing. Informational constraints will be primarily targeted to applications that load and analyze data that originated from a data warehouse. For such applications, the conditions for a given constraint are known to be true, so the constraint does not need to be enforced during data load operations. This session will cover the support for primary and foreign key (referential integrity) constraints in Spark. You’ll learn about the constraint specification, metastore storage, constraint validation and maintenance. You’ll also see examples of query optimizations that utilize referential integrity constraints, such as Join and Distinct elimination and Star Schema detection.

Informational Referential Integrity Constraints Support in Apache Spark with ...

Databricks

Big Telco - Yousun Jeong

Spark Summit

A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)

Spark Summit

Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on row based columnar datasets through full scan but does not provide constructs for column indexing and time series analysis. For dealing with document datasets with timestamps where the features are represented as variable number of columns in each document and use-cases demand searching over columns and time to retrieve documents to generate learning models in realtime, a close integration within Spark and Lucene was needed. We introduced LuceneDAO in Spark Summit Europe 2016 to build distributed lucene shards from data frame but the time series attributes were not part of the data model. In this talk we present our extension to LuceneDAO to maintain time stamps with document-term view for search and allow time filters. Lucene shards maintain the time aware document-term view for search and vector space representation for machine learning pipelines. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms with filters on time. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. Our synchronous API uses Spark-as-a-Service to power analytical queries while our asynchronous API uses kafka, spark streaming and HBase to power time series prediction algorithms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms and configurable time stamp aggregate columns. We will demonstrate the latency of APIs on a suite of queries generated from terms. Key takeaways from the talk will be a thorough understanding of how to make Lucene powered time aware search a first class citizen in Spark to build interactive analytical query processing and time series prediction algorithms.

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Spark Summit

Prototypes are typically re-implemented in another language due to compatibility issues with R in the enterprise, but TIBCO Enterprise Runtime for R (TERR) allows the language to be run on several platforms. Enterprise-level scalability has been brought to the R language, enabling rapid iteration without the need to recode, re-implement and test. This presentation will delve further into these topics, highlighting specific use cases and the true value that can be gained from utilizing R. The session will be followed by a lively, open Q&A discussion.

Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...

Data Con LA

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Building a Big Data Pipeline

Jesus Rodriguez

Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee

Spark Summit

Hadoop application architectures - using Customer 360 as an example

hadooparchbook

Realtime streaming architecture in INFINARIO

Jozo Kovac

Technical Leads and Databricks Champions Darren Fuller & Sandy May will give a fast paced view of how they have productionised Data Quality Pipelines across multiple enterprise customers. Their vision to empower business decisions on data remediation actions and self healing of Data Pipelines led them to build a library of Data Quality rule templates and accompanying reporting Data Model and PowerBI reports. With the drive for more and more intelligence driven from the Lake and less from the Warehouse, also known as the Lakehouse pattern, Data Quality at the Lake layer becomes pivotal. Tools like Delta Lake become building blocks for Data Quality with Schema protection and simple column checking, however, for larger customers they often do not go far enough. Notebooks will be shown in quick fire demos how Spark can be leverage at point of Staging or Curation to apply rules over data. Expect to see simple rules such as Net sales = Gross sales + Tax, or values existing with in a list. As well as complex rules such as validation of statistical distributions and complex pattern matching. Ending with a quick view into future work in the realm of Data Compliance for PII data with generations of rules using regex patterns and Machine Learning rules based on transfer learning.

Building Data Quality pipelines with Apache Spark and Delta Lake

Databricks

TechEvent Databricks on Azure

Trivadis

The challenge of computing big data for evolving digital business processes demands variety of computation techniques and engines (SQL, OLAP, time-series, graph, document store), but working in unified framework. A simple architecture of data transformations while ensuring the security, governance, and operational administration are the necessary critical components for enterprise production environments supporting day-to-day business processes. In this session, you will learn about best practices & critical components to ensure business value from latest production deployments. Hear how existing customers are using SAP Vora and the value they have achieved so far with this in-memory engine for distributed data processing. The session provides you with a clear understanding how SAP Vora and open source components like Apache Hadoop and Apache Spark offer an architecture that supports a wide variety of use cases and industries. You will also receive very useful insight where to find development resources, test drive demos, and general documentation.

Modernizing Business Processes with Big Data: Real-World Use Cases for Produc...

DataWorks Summit/Hadoop Summit

In this talk, we will explore how Uber enables rapid experimentation of machine learning models and optimization algorithms through the Uber’s Data Science Workbench (DSW). DSW covers a series of stages in data scientists’ workflow including data exploration, feature engineering, machine learning model training, testing and production deployment. DSW provides interactive notebooks for multiple languages with on-demand resource allocation and share their works through community features. It also has support for notebooks and intelligent applications backed by spark job servers. Deep learning applications based on TensorFlow and Torch can be brought into DSW smoothly where resources management is taken care of by the system. The environment in DSW is customizable where users can bring their own libraries and frameworks. Moreover, DSW provides support for Shiny and Python dashboards as well as many other in-house visualization and mapping tools. In the second part of this talk, we will explore the use cases where custom machine learning models developed in DSW are productionized within the platform. Uber applies Machine learning extensively to solve some hard problems. Some use cases include calculating the right prices for rides in over 600 cities and applying NLP technologies to customer feedbacks to offer safe rides and reduce support costs. We will look at various options evaluated for productionizing custom models (server based and serverless). We will also look at how DSW integrates into the larger Uber’s ML ecosystem, e.g. model/feature stores and other ML tools, to realize the vision of a complete ML platform for Uber.

Building Intelligent Applications, Experimental ML with Uber’s Data Science W...

Databricks

High-Scale Entity Resolution in Hadoop

DataWorks Summit/Hadoop Summit

Data infrastructure architecture for medium size organization: tips for colle...

DataWorks Summit/Hadoop Summit

Spark Summit East 2015 Keynote -- Databricks CEO Ion Stoica

Databricks

Druid Overview by Rachel Pedreschi

Brian Olsen

Spark - Migration Story

Roman Chukh

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Data Con LA

What's hot (20)

Informational Referential Integrity Constraints Support in Apache Spark with ...

Big Telco - Yousun Jeong

A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Big Data Day LA 2016/ Big Data Track - Apply R in Enterprise Applications, Lo...

Spark in the Enterprise - 2 Years Later by Alan Saldich

Building a Big Data Pipeline

Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee

Hadoop application architectures - using Customer 360 as an example

Realtime streaming architecture in INFINARIO

Building Data Quality pipelines with Apache Spark and Delta Lake

TechEvent Databricks on Azure

Modernizing Business Processes with Big Data: Real-World Use Cases for Produc...

Building Intelligent Applications, Experimental ML with Uber’s Data Science W...

High-Scale Entity Resolution in Hadoop

Data infrastructure architecture for medium size organization: tips for colle...

Spark Summit East 2015 Keynote -- Databricks CEO Ion Stoica

Druid Overview by Rachel Pedreschi

Spark - Migration Story

Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...

Viewers also liked

Architecting a Next Generation Data Platform

hadooparchbook

Churn management

Mohammed Akram Ayyubi

Presentación Gregorio Trimarco | Mastercard - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Deb Reyes | Google - eCommerce Day Buenos Aires 2017 12 30 4. de...

eCommerce Institute

Presentación Juan Tomac | Unilever - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Juan Pablo Lafosse | Almundo - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Mariano Tordo, Farmacity & Andres Zaied, Musimundo - eCommerce D...

eCommerce Institute

Presentación Jorgelina Striedinger | Digital Element - eCommerce Day Buenos A...

eCommerce Institute

Presentación Alberto Banano Pardo | AdsMovil - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Cristian Adamo | Avantrip - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentaciones Gustavo Sambucetti | CACE & GoforEcommerce - eCommerce Day Bue...

eCommerce Institute

Presentación Sergio Grinbaum | Think Thanks - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Eliane Iwasaki | Return Path - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Joan Miró | NetQuest - eCommerce Day Buenos Aires 2017

eCommerce Institute

Presentación Francisco Berroeta | Samsonite - eCommerce Day Buenos Aires 2017

eCommerce Institute

Apache Spark is one of the most popular big data projects, offering greatly improved performance over traditional MapReduce models. Much of Apache Spark’s power comes from lazy evaluation along with intelligent pipelining, which can make debugging more challenging. Holden Karau and Joey Echeverria explore how to debug Apache Spark applications, the different options for logging in Spark’s variety of supported languages, and some common errors and how to detect them. Spark’s own internal logging can often be quite verbose. Holden and Joey demonstrate how to effectively search logs from Apache Spark to spot common problems and discuss options for logging from within your program itself. Spark’s accumulators have gotten a bad rap because of how they interact in the event of cache misses or partial recomputes, but Holden and Joey look at how to effectively use Spark’s current accumulators for debugging before gazing into the future to see the data property type accumulators that may be coming to Spark in future versions. And in addition to reading logs and instrumenting your program with accumulators, Spark’s UI can be of great help for quickly detecting certain types of problems. Holden and Joey cover how to quickly use the UI to figure out if certain types of issues are occurring in our job.

Debugging Apache Spark - Scala & Python super happy fun times 2017

Holden Karau

Viewers also liked (16)