2. ML @ dropbox
Our signal sources:
Files
Multi-exabyte data
File Metadata
Trillions
User interactions
Billions / day
3. ML @ dropbox
ML Impact at Dropbox:
โ Smart Sync
โ Content Suggestions
โ Team Activity Ranking
โ Search Ranking
โ OCR
And many more โฆ
4. ML Platform
Challenges:
โ Huge data sources that are isolated in various system across production
โ Multiple privacy levels of data
โ Custom work and build dedicated services for each new use case
โ Manual training which is hard to reproduce
โ Wide variety of development processes and ML frameworks
5. ML Platform
Mission:
Accelerate intelligent product development at Dropbox
By:
โ Scalable access to data for offline and online
โ Ensures sensitive data is protected and accessed only in approved ways
โ Easy model deployment & experimentation
โ Automate workflows
โ Standardize the process, frameworks and tools
8. Antenna
What is Antenna?
โ User activity service
โ Provides various ways to query activity events
โ Support aggregations for simple summaries and histograms of activity data
15. Data Preparation - Predict logger
- Converting raw logs into labeled datasets
- Logging partial information from different services at different times
- Eliminate discrepancies between online and offline
17. Prototyping
HDFS
Signal and training
data store
Spark
Zeppelin Notebooks
Multi-user notebook environment
Workbench
40 cores, 400GB ram
dbxlearn
Elastic ML training and
hyperparameter tuning
18. dbxlearn
What is dbxlearn?
โ dbxlearn provides an easy way to use computing at scale for training
โ Core problems dbxlearn is addressing:
โ Elasticity
โ Standard way to train on different hw configurations (GPU, TPU) on
different cloud platforms.
โ Hybrid cloud architecture - Interface with private cluster and well as public
clouds
โ Currently integrated with AWS and use SageMaker
24. Shadow experimentation - Suggest backend
โ Send live traffic to shadow cluster with a different experiment variant
โ Results are logged for experiment analysis
โ Useful to collect labeled datasets using Predict Logger
28. Summary
โ End-to-End platform that supports all steps in ML development
workflow
โ Deep integration with Dropbox large scale data sources
โ Flexible APIs to support wide variety of use cases
โ Hybrid cloud architecture for elasticity and early adoption of new
technologies
29. Next Challenges
โ Better representation of data relations across multiple systems
โ Democratize ML at dropbox, extending our tools from ML
developers to more engineers
Understand the scale of our data sources that we use for ml features, not necessarily for training
huge file repository. probably one of the largest in the world. containing exabytes of data with millions of new files added every day.
And we have file system trees which provides our systems valuable signals on content organization and grouping.
billions of events every day of the users interaction with these files.
Which provide our systems valuable signals on how users are working with files and collaborate with one another in their workplace.
These are huge data sources that present lots of potential for ML, but also lots of challenges to ML developers.
Go through some of the use cases
Smart sync is trying to predict which files you will need on each device so we can make it available locally and vacuum unneeded files.
Content suggestion is trying to simplify retrieval process and predict which file you are looking for
Team Activity ranking is trying to increase awareness of what others are doing while filtering the noise
Search ranking - use a ml model for ranking search results
OCR - we extract text from each image which is uploaded to dropbox
So these are some of the highlights, but there are many more.
So what are the challenges that we are trying to solve?
Our huge data sources are isolated in various system across production, which makes it challenging to access for training.
Multiple privacy levels of data, some of which is not reviewable by engineers
Teams are doing custom work and build new services for solve their problems
There are manual complex training workflows which are hard to reproduce
Wide variety of ml frameworks and tools in use across teams
So Our mission: Accelerate intelligence products development at Dropbox
And we trying to solve it by:
provide scalable access to context to models both offline and online
Ensures sensitive data is protected and accessed only in approved ways
Make it easy to deploy models without building new services and make it easy to experiment with new models
Automate development workflows
Standardize the process, frameworks and tools for Intelligent product development and release
All ml development has a common basic workflow that includes: Data collection, Data preparation, Training & evaluation, And model deployment to production.
We have developed components to support each step in this basic workflow.
Online components integrates with DBX production systems to provide data in realtime. And make it easy to deploy and experiment with new models.
Our offline components capture historical data and make it easy to access it for training and prototyping.
And Management components are making it easy to automate these workflows.
Serves user activity to online production services
Given a user what are all the all the files that the user have interacted on. Given a file who are all the users that have interacted with that file.
We also support simple aggregations like how many edits a user had on a specific file or creating histograms of number of events across different days of week
In this example we are creating suggestions for a user that try to predict which file the user will open next
To do that it will first query Antenna our user activity service for the user activity from the last few months.
It will then use that to generate a candidate list of files for suggestions and aggregations of the user activity on each file candidate.
Then an ml model will rank them and return the top N.
It is important that the data will be fresh. The user recent activity, even from the last few minutes, is relevant for what the user want to do next.
Online ingestion hosts will process events in realtime , updating indexes and aggregations and provide fresh data.
Offline components persist the raw events in a durable store and rebuild the indexes and aggregations periodically
ml developers can define new aggregators and indexes that will be run in both in online ingestion and in offline workers and get automatically backfilled
Antenna is our ingestion infrastructure for user activity, lets now talk about content ingestion infrastructure
OCR is a good example that will demonstrate how an ML model will run in our content ingestion infrastructure
It is actually a multistep process of not one but several models .
Every image that passes through our ingestion pipeline
classified whether it contain OCR-able content,
image is rectified to align the text ,
deep net model is use to extract word boxes,
LSTM model is used to convert each word box to a sequence of characters,
and finally a lexicon based algorithm is converting these sequence of characters to actual words.
Every File update in ingested by our indexers that call a plugin framework to do a transform on the user content, which is in our case an OCR model,
and then it store the results in Doc Store. Which contain all derived data for each file.
When running a transform on raw user content, there are security concerns for exploits and vulnerabilities, so the plugin framework run each plugin in a sandboxed environment we call jail.
This can allow us to be sure that any exploits in the frameworks that we are using, like ImageMagick and TF, cannot be used to to gain access to our systems.
One of the challenges that we are dealing with here is how to simplify model deployment to this jailed environment and enable easy experimentation with different model variants.
Our ETL pipeline makes it easy to generate training data and signals from our data sources
We maintain interfaces for spark jobs to import data from our data lake and Antenna
Using periodical Spark jobs orchestrated by Airflow to generate signals and training data
Then ML developer can access the output signal and training data for training.
We provide a more specialized pipeline that help automate the generation of labeled datasets from live traffic.
Which we call the predict logger
The predict logger define an api with a set of predict events that define the life cycle of online predictions
Like : requested, predicted, viewed, actions
This events are being logged from different services at different time. And the predict logger merge them in a consistent way that help developers avoid incidental complexities
The result labeled dataset is ready for use in training with all the signals and context as was seen in serving time, so this help us avoid the discrepancies between online and offline data
Before full scale training, developers first need to prototype
For prototyping they need access to training and signal data from the etl pipeline which they use for exploration and offline evaluations
We use workbenches in production that are integretage with a Spark cluster to provide them access to all the offline data
After prototyping, they use dbxlearn for large scale training and hyperparameter tuning.
Training jobs require lots of computing power
dbxlearn provides our developers an easy way to use computing at scale for training by enabling them to submit training jobs to remote clusters.
It provides us elasticy, make sure that each job will get all the resources that it needs when it needs them
Also a standard way to train on different hw configuration, so we can use specialized hw configuration in training like GPUs.
We have build a hybrid cloud architecture that interface with our private cluster as well as public clouds.
We have integrated currently with AWS and use SageMaker for training.
Typical workflow with dbxlearn
Use dbxlearn train --local to test the training script locally
Remove --local to test the training script in the cloud.
If it runs well in the cloud, use dbxlearn tune to find the optimal hyperparameters automatically
Use dbxlearn query to check the status and the results of all training jobs
If the results are good, use dbxlearn deploy-model to deploy the best model to the model store
We have a central service to host models in production called predict service
Loads models from the model store and provide an standard API to do realtime inference.
Support multiple model inference partitions for resources isolation.
The inference api can also provide a proxy for running inference on public cloud services
Help reduce boilerplate of running live experiments
Simple config defines which signals to collect and which model to run .
Client send requests with target experiment variant
Suggest backend run the signal collector that was defined in the config
And then run configured model
Standard logging to monitor experiments results
Provide signal collector abstraction that developers can customize
The example is a campaign ranker that was implemented as a contextual multi-arm bandits problem
At dropbox we have a campaign framework that makes it easy to define campaigns that can be displayed in various ui surfaces for various populations.
In this case one of our web pages is displaying a campaign for Dropbox business.
For each impression there are many competing campaigns, that can be modeled as a multi-arm bandits problem, where we need to choose which arm to play.
And we use ui features and user features as context for this decision.
Impressions are being logged by UI surfaces and by predict service from the backend, using the predict logger
This generates a batch of new training data periodically by our ETL pipeline
A training job then update the policies in the contextual bandits model and store it in the model store
The predict service then load the new model and use it for future decisions
And this repeats itself forever
This demonstrate the use of our ETL pipeline to automate the full workflow of labeled data generation, training and model deployment
We have built an End-to-End platform that supports all steps in ML development workflow
We provide Deep integration with Dropbox large scale data sources to make this data accessible for offline training and online inference
We built our components with flexible APIs to support wide variety of use cases
We chose a hybrid cloud architecture for easier elasticity and early adoption of new technologies
Currently the exabytes of data are in multiple systems. But there are relationships that would be useful to know across these systems. So our challenge is how can we present the interaction among data across systems so the ML team can use them for features?
Currently our tools are used mainly by ml developers with high ML expertise. We would like to make our tools accessible to more engineers with better and simpler interfaces
