Roblox is a global online platform bringing millions of people together through play, with over 37 million daily active users and millions of games on the platform. Machine learning is a key part of our ability to scale important services to our massive community. In this talk, we share our journey of scaling our deep learning text classifiers to process 50k+ requests per second at latencies under 20ms. We will share how we were able to not only make BERT fast enough for our users, but also economical enough to run in production at a manageable cost on CPU. Further details can be found in our blog post below: https://robloxtechblog.com/how-we-scaled-bert-to-serve-1-billion-daily-requests-on-cpus-d99be090db26

1. How We Scaled BERT
To Serve 1+ Billion Daily
Requests on CPU
Quoc N. Le, Data Scientist, Roblox
Kip Kaehler, Engineering Manager, Roblox

2. About Roblox
3

3. Deep Learning for Text Classification
4
● Text Classification is a key capability on
Roblox platform
● BERT is a deep learning model that
has transformed the Natural Language
Processing (NLP) landscape
● Performance (Precision/Recall Area
Under Curve) of our text classifiers
improved by 10 percentage points fine-
tuning BERT versus classical machine
learning

4. Beyond Accuracy: Latency and Throughput
5
Latency: Speed of Request
Analogy -> How long it takes for a
single person to cross a bridge
We required latency under 20ms
Throughput: Completed Requests
Per Second
Analogy -> How many people can
cross a bridge in a period of time
We required over 50k requests per sec
We want a short, wide bridge to
maximize both Latency and
Throughput in a realtime
environment

5. Our Deep Learning Tech Stack

6. GPU vs CPU (for our application)
Higher Throughput
for Real-Time Inference
Higher Throughput
for Model Training
~GPU (TESLA V100) 10x faster
at processing training examples
than CPU, due to efficiency in
doing large batch matrix
operations
OR
~CPU (Intel Xeon Scalable
Processor) 5x more throughput
than GPU due to CPU-specific
optimizations and spreading real-
time inference requests across
cores (with latency < 20ms)
(on cost equivalent hardware in 2020)

7. Which Comes First?
8
Build an Accurate
Model
Make It Fast
Choose a Known
Fast Model
Make It Accurate

8. Know Your Quoc Le’s
Quoc N. Le Quoc V. Le
Has over 85k citations
as an AI researcher
according to Google
Scholar
Once got kicked out of
the Boomtown Casino
in Reno for counting
cards in blackjack

9. Our Scaling Playbook on CPU: Less Is More!
10
❏ Smaller Model (Distillation)
❏ Smaller Inputs (Dynamic Inputs)
❏ Smaller Weights (Quantization)
❏ Smaller Number of Requests
(Caching)
❏ Smaller Number of Threads per
Core (Thread Tuning)

10. Where We Started
Smaller
Model
Smaller
Inputs
Smaller
Weights
Benchmarks
Run on Intel
Xeon
Scalable
Processors
Baseline
BERT

11. Smaller Model (DistilBERT)
Smaller
Model
Smaller
Inputs
Smaller
Weights
Benchmarks
Run on Intel
Xeon
Scalable
Processors
Baseline
BERT

12. Smaller Model (DistilBERT)
Bert Base has
110m parameters
DistilBert has
66m parameters
Tradeoff: < 1%
negative impact on
“Accuracy” (PR
AUC)
Text Input
..
..
..
..
..
Predict
Layer
Transformer Layer
..
..
..
..
Transformer Layer
Transformer Layer
..
..
..
..
Transformer Layer
Predict
Layer
Text Input
Student
Model
(DistilBERT)
Teacher
Model
(BERT)
DistilBERT: https://arxiv.org/pdf/1910.01108.pdf
..
Knowledge
Distillation

13. Smaller Inputs (Dynamic Shapes)
Smaller
Model
Smaller
Inputs
Smaller
Weights
Benchmarks
Run on Intel
Xeon
Scalable
Processors
Baseline
BERT

14. Smaller Inputs (Dynamic Shapes)
Fixed Shape Inputs
(Zero pad until all inputs have same shape)
Dynamic Shape Inputs
(Do not zero pad)

15. Smaller Weights (Quantization)
Smaller
Model
Smaller
Inputs
Smaller
Weights
Benchmarks
Run on Intel
Xeon
Scalable
Processors
Baseline
BERT

16. Smaller Weights (Quantization)
Image Credit:
https://towardsdatascience.com/how-to-
accelerate-and-compress-neural-networks-with-
quantization-edfbbabb6af7
Tradeoff: < 1%
negative impact on
“Accuracy” (PR
AUC)
“Dynamic Quantization” One-liner

17. Smaller Weights (Quantization)

18. Smaller Number of Requests to Model (Caching)
Image Credit:
https://peltarion.com/blog/data-
science/illustration-3d-bert
Text
Classification
Service
Cache
DistilBERT Model
1. Retrieve text
classification result
from cache (we’re done
if it’s there)
2. Else call deep
learning model for
result
3. Add result to cache,
then return result to
service

19. Smaller Number Threads Per Core (Thread Tuning)

20. Our Scaling Playbook on CPU: Less Is More!
21
✓ Smaller Model (Distillation)
✓ Smaller Inputs (Dynamic Inputs)
✓ Smaller Weights (Quantization)
✓ Smaller Number of Requests
(Caching)
✓ Smaller Number of Threads per
Core (Thread Tuning)

21. 30x Improvement in Latency and Throughput on CPU
Smaller
Model
Smaller
Inputs
Smaller
Weights
Benchmarks
Run on Intel
Xeon
Scalable
Processors
Baseline
BERT

22. Takeaways
23
● For certain real-time deep learning
applications, it is feasible/natural to super-
scale inferences on CPU
● The key to scaling is making things smaller,
as shown in this presentation
● Many optimizations that enabled scale are
easy to implement (one-liners)
● Check out our blog for more details:
https://robloxtechblog.com/how-we-scaled-
bert-to-serve-1-billion-daily-requests-on-cpus-
d99be090db26

23. Questions? Suggestions?
24
We are always looking to get more performance
from our models. Please reach out to
kkaehler@roblox.com
PS We are always hiring 🤓

24. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.