This document discusses lessons learned from Intuit's experience deploying machine learning models with AWS managed services. The key lessons are: (1) work with vendors to implement necessary security measures; (2) add your own monitoring and tooling; (3) make vendors aware of any issues found; and (4) be transparent about the costs of hosting models. Intuit found that implementing these lessons improved security, helped identify performance issues, and increased transparency into model hosting costs.
Al Barsha Escorts $#$ O565212860 $#$ Escort Service In Al Barsha
Intuit's Machine Learning Platform Journey
1. Intuit Proprietary 1
Mar 25, 2019
Practical Lessons on Model
Deployments with AWS
Managed Services
Ian Sebanja
Tobias Wenzel
2.
3. “We spend more time bringing the model to
production than developing and training it”
Data Science Operations is not easy
4. “We spend more time bringing the model to
production than developing and training it”
Data Science Operations is not easy
5. Model Development Lifecycle
Deploy / HostPerf TestTraining
Model
Development
/ Training
Feature
Pipelines
Explore
Intuit ML Platform
Cloud
Swarm
Sagemaker
Sagemaker
6. Intuit’s ML Platform
Services Layer
Prediction
Logging
Data
Aggregation
Model Deployment
Data
Prefetching
Logging & Monitoring
Model Hosting
MLPlatform
Self-Service
28. 1. Work with the vendor to get the security you need
Lessons learned
29. 1. Work with the vendor to get the security you need
- Monitor traffic from endpoints hosting a model
- Set security groups to block traffic where necessary
- Trace requests through all layers of the platform
Lessons learned
30. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
Lessons learned
31. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
- Helped greatly during performance tests of the service
- Find breaking points early - collect your own data
- Test out uncommon usages - 1 request per minute?
- Create automation around right-sizing infrastructure for models
Lessons learned
32. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
3. Make the vendor aware of everything
Lessons learned
33. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
3. Make the vendor aware of everything
- Monitoring and alerting can reveal unintended behaviour
Lessons learned
34. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
3. Make the vendor aware of everything
4. Be very transparent about cost of a model
Lessons learned
35. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
3. Make the vendor aware of everything
4. Be very transparent about cost of a model
- Intuit implemented a “billback” mechanism
Lessons learned
36. 1. Work with the vendor to get the security you need
2. Add your own monitoring and tooling
3. Make the vendor aware of everything
4. Be very transparent about cost of a model
Lessons learned