In early September, Apple released a paper describing Overton, the framework they built to create, monitor, and improve production-based ML systems. After presenting the main lines that define this framework, we will take a closer look at the heart of Overton: slice-based learning.

1. OvertonOverton
Apple- avored MLApple- avored ML
2019/10/07 Nantes Machine Learning Meetup2019/10/07 Nantes Machine Learning Meetup

2. OverviewOverview

3. OverviewOverview
What / When / WhereWhat / When / Where
Publishing date
Early September
Conference
NeurIPS
Goal
ML software lifecycle management
Maturity
Production

4. OverviewOverview
Challenges to overcomeChallenges to overcome
Precise monitoring
Complex pipelines
Ef cient feedback loop

5. OverviewOverview
Architectural choicesArchitectural choices
Code-less deep learning
Multi-task learning
Weak supervision

6. Code-less deepCode-less deep
learninglearning

7. Code-less deep learningCode-less deep learning
PrinciplesPrinciples
Models & training code = experts
“black box” by engineers

8. Code-less deep learningCode-less deep learning
ModularityModularity

9. Code-less deep learningCode-less deep learning
Con gurationCon guration

10. Fine grained MLFine grained ML
& multi-tasks& multi-tasks

11. Fine grained ML & multi-tasksFine grained ML & multi-tasks
ProblemsProblems
Lots of subtasks (implicit & explicit)
Need to evaluate & monitor them
Need to improve on them

12. Fine grained ML & multi-tasksFine grained ML & multi-tasks
ApproachApproach
Slice-based learning
De nition of data subsets
Augmentation of model capacity
Dedicated metrics

13. Fine grained ML & multi-tasksFine grained ML & multi-tasks
Slice-based learningSlice-based learning

14. Fine grained ML & multi-tasksFine grained ML & multi-tasks
De nition of critical dataDe nition of critical data
subsetssubsets
With “slice functions”:
def sf_bike(x):
return "bike" in object_detector(x)
def sf_night(x):
return avg(X.pixels.intensity) < 0.3

15. Fine grained ML & multi-tasksFine grained ML & multi-tasks
Slice expertsSlice experts
For each slice, an expert:
should add capacity to the model
has to know when to trigger
has to have dedicated metrics

16. Fine grained ML & multi-tasksFine grained ML & multi-tasks
Hard partsHard parts
Noise : slices are de ned with heuristics
Scale 1 : when the number of slices goes up, does the
model still run fast enough?
Scale 2 : when the number of slices goes up, is the
model still good enough?

17. Fine grained ML & multi-tasksFine grained ML & multi-tasks
Proposed solutionProposed solution

18. ResourcesResources
Snorkel blog “Slice-based Learning”
NeurIPS paper “Slice-based Learning: A
Programming Model for Residual Learning in Critical
Data Slices”

19. WeakWeak
supervisionsupervision

20. Weak supervisionWeak supervision
Problem statementProblem statement
We need data, but:
It's expensive, not available, yadda yadda
Especially for interesting corner cases
It's noisy

21. Weak supervisionWeak supervision
SolutionSolution
Use heuristics
@labeling_function()
def lf_regex_check_out(x):
"""Spam comments say 'check out my video', 'check it out', et
return SPAM if re.search(r"check.*out", x.text, flags=re.I) e

22. Weak supervisionWeak supervision
SolutionSolution
Then correct the loss accounting for their correlation

23. Weak supervisionWeak supervision
ImplementationImplementation
Overton uses a modi ed version of the “Label Model”
from Snorkel.

24. ResourcesResources
Snorkel blog « Introducing the New Snorkel »
ICML paper “Learning Dependency Structures for
Weak Supervision Models”

25. ConclusionConclusion
Modularity (alike AllenNLP, tensor2tensor)
Dedicated metrics & multi-tasks
Improvements on critical data subsets
Arti cal data with theoretical corrections

26. Questions /Questions /
DiscussionDiscussion