Autopilot: workload autoscaling at Google

1. Autopilot: workload
autoscaling at Google
Krzysztof Rzadca (Google & University of Warsaw, Poland), Paweł Findeisen, Jacek Świderski, Przemysław Zych,
Przemek Broniek, Jarek Kuśmierek, Paweł Nowak, Beata Strack, Piotr Witusowski, Steven Hand, John Wilkes
(Google)
EuroSys 2020
April 2020

2. Proprietary + Confidential
Google runs in
containers
In any given week, we
launch over two billion
containers across
Google.

3. Proprietary + Confidential
Resource limits are crucial to isolate workloads
container limit:
max amount of CPU/mem
a container can use
container usage:
CPU/mem used
container slack:
CPU/mem wasted

4. Proprietary + Confidential
Borg, our scheduler,
packs containers to
machines by resource
limits.
image source: http://dx.doi.org/10.1145/2741948.2741964 [Verma et al., EuroSys’15]
machines

5. Proprietary + Confidential
Limits are fine-grained:
CPU in milli-cores
memory in bytes
Source: http://dx.doi.org/10.1145/2741948.2741964 [Verma et al., EuroSys’15]

6. Proprietary + Confidential
We pack containers to machines by limits.
So, precise limits are crucial for efficiency and reliability.
limit
machine
capacity
container B
container C
container A

7. Proprietary + Confidential
We pack containers to machines by limits.
So, precise limits are crucial for efficiency and reliability.
precise
limits
good!
limit
limit
usage
machine
capacity
container B
container C
container A

8. Proprietary + Confidential
We pack containers to machines by limits.
So, precise limits are crucial for efficiency and reliability.
out-of-resources
crash
precise
limits
good!
limit
resource are
wasted
(underallocated
machine)
bad! bad!
limit
usage
requested
usage
container
limit
machine
capacity
container B
container C
container A

9. Proprietary + Confidential
Autopilot acts as a controller for Borg limits.
Autopilot continuously adjusts resource limits:
CPU/Mem limits for containers (vertical scaling),
number of replicas (horizontal scaling).
container
limits
container
counts container
limits
start/stop
containers

10. Proprietary + Confidential
Autopilot Recommenders

11. Proprietary + Confidential
Moving window recommenders
● Exponentially-decaying samples
(half-life of 48 hours)
● Compute statistics over the
samples, e.g. 95%ile
● add a safety margin
time
resources
usage
98%ile

12. Proprietary + Confidential
Moving window recommenders
● Exponentially-decaying samples
(half-life of 48 hours)
● Compute statistics over the
samples, e.g. 95%ile
● add a safety margin
time
resources
usage
exponential decay
98%ile

13. Proprietary + Confidential
Moving window recommenders
● Exponentially-decaying samples
(half-life of 48 hours)
● Compute statistics over the
samples, e.g. 95%ile
● add a safety margin
time
resources
usage
exponential decay
safety margin
limit

14. Proprietary + Confidential
● Each model is an arg-max
algorithm picking a limit value
● Each model is parametrized by
the decay rate and the safety
margin.
● The recommender picks the
model performing the best over a
longer time period.
Machine learning recommenders
decay rate
limit
model n
model 1 model 2 …….

15. Proprietary + Confidential
Evaluation:
Observational study of production jobs
Focus on memory

16. Proprietary + Confidential
Autopilot efficiency - reduction of slack
relative slack:
(av_limit - 95%ile usage) / (av_limit)
16
limit(t)
usage(t)
slack(t)
absolute slack:
∫ slack(t) dt = ∫ limit(t) dt - ∫ usage(t) dt
unit: capacity of a single (largish)
machine
av_limit
average
limit during
the day
95%ile
usage
during
the day
(av_limit - 95%ile usage)

17. Proprietary + Confidential
A random sample of 5000 jobs in each
category.
Cumulative
distribution
function
Autopiloted jobs have
significantly smaller
relative slack.
Relative slack: (av_limit - 95%ile usage) / av_limit
av_limit
(av_limit - 95%ile usage)
better worse

18. Proprietary + Confidential
Autopiloted jobs save
significant capacity.
Cumulative
distribution
function
Absolute slack [machines]
Absolute slack
A random sample of 5000 jobs in each
category.
better
worse

19. Proprietary + Confidential
When jobs migrate to
Autopilot, their slack is
significantly reduced.
A random sample of 500
jobs that migrated to
autopilot in a certain
month, m0.
CDFs for slack for 2 months
before and after migration
reduction of relative slack
Cumulative
distribution
function
better
worse

20. Proprietary + Confidential
Autopilot Reliability:
how frequent are
out-of-memory errors.
We count terminations of
containers.
We weight the number of
terminations by the average
number of containers of a job.
out-of-resources crash
requested
usage
container
limit
machine
capacity

21. Proprietary + Confidential
Autopilot reduces the
frequency of
out-of-memory events.
OOMs are rare: 99.5% of
autopiloted jobs have no
OOMs.
Cumulative
distribution
function
better
worse

22. Proprietary + Confidential
DevOps:
Autopiloted jobs account for over
48% of Google’s fleet-wide resource
usage.

23. Proprietary + Confidential
Autopilot’s dynamic limits could help to
keep the job running despite bugs.

24. Proprietary + Confidential
1. Efficient scheduling requires fine-grained control of jobs’ limits
2. Humans are bad at setting the limits precisely.
3. Autopilot uses past usage to drive future limits
4. Autopilot reduces relative slack by 2x
...and it reduces the number of jobs severely impacted by OOMs 10x
Autopilot: workload autoscaling at Google