This document discusses machine learning models in production environments. It begins with an overview of developing a machine learning model, from data collection and analysis to model training and validation. It then discusses issues that can arise with models in production, such as data or regulatory changes causing model drift over time. Finally, it discusses best practices for machine learning operations (MLOps), including continuous monitoring of models, versioning of code, data and models, testing throughout the development process, and continuous delivery and retraining of models.

1. DIRECTION GÉNÉRALE DU SYSTÈME D’INFORMATION
IATSUN IANA
PROJECT MANAGER FOR APPLICATIONS WITH
ARTIFICIAL INTELLIGENCE
2022
MACHINE LEARNING IN PRODUCTION:
FROM DATA-SCIENTIST TO THE FINAL
USER

2. Banque de France, all rights reserved
PERSONAL EXPERIENCE
La Reunion

3. Banque de France, all rights reserved
PERSONAL EXPERIENCE
Overcrowded hall Waiting Time from this point is 2h

4. Banque de France, all rights reserved
BRIEF PRESENTATION
Since 2020 Project manager for applications with Artificial
intelligence at Banque de France
2016-2020 R&D engineer at IDEMIA ( Safran Identity and
Security)
2015-2016 R&D engineer at Nestor Technologies
2011-2014 PhD in image and signal processing at Poitiers
University
2009-2011 MS in electronics at National Technical University
of Ukraine “Kyiv Polytechnic Institute”
IATSUN IANA

5. Banque de France, all rights reserved
Missions of the Banque de France
 Monetary strategy
 Guardian of the currency
 Financial stability
 Economic services
BANQUE DE FRANCE
The Banque de France is
the national central bank and
the French pillar of the Eurosystem

6. Banque de France, all rights reserved
DEVELOPING MACHINE LEARNING MODEL
Data collection
(images with faces)
Problem scope
(ex. Face recognition)
Evaluation
and
validation of
model
Data
Preparation
Model
training
Data Analysis
Face recognition ML model
Hyper-
parameters
adjustment
NB : Face recognition is an example, Banque de France does not conduct this study

7. FACE RECOGNITION LEARNING
Angelina JOLIE
Thousands of people
class 1 : person 1
class 2 : Angelina JOLIE
class X : person X
Jennifer ANISTON
Banque de France, all rights reserved

8. FACE RECOGNITION IN USE
Sarah Jessica PARKER
Jennifer ANISTON
Kate MIDDLETON
Banque de France, all rights reserved

9. Use
Learning
WORLD IS IN PERMANENT CHANGE
Banque de France, all rights reserved
0001 Angelina JOLIE
0002 Jennifer ANISTON
0003 Courteney COX
Historical data Future data

10. FIRST SIGN OF PERFORMANCE DRIFT
Banque de France, all rights reserved
December 2019
September 2020

11. Banque de France, all rights reserved
MODEL IN PRODUCTION DRIFT
Model drift = red light

12. Banque de France, all rights reserved
MACHINE LEARNING MODEL LIFE IN PRODUCTION
Data in production
Monitoring Dashboard Regular audit
Regulation changes
ML model
Unexpected problems in
production
White book « Y A-T-IL UNE VIE APRÈS LES POCs ? RÉUSSIR l’INDUSTRIALISATION DU BIG DATA » par Quantmetry
Software

13. Banque de France, all rights reserved
Article Machine Learning in Production (MLOps)
PART OF MACHINE LEARNING CODE
Configuration
Data Collection
Feature extraction
ML code
Data verification
Analysis Tool
Process Management Tool
Machine Ressource
Management
Serving
Infrastructure
Monitoring

14. Banque de France, all rights reserved
MACHINE LEARNING PROJECT LIFECYCLE
Scope the
project
Define and
explore
the data
Organise
and
analyse
the data
Train the
model,
optimise
parameters
Validate
the model
Deploy in
production
Monitor &
Maintain
the system
Data Modeling Deployment

15. Machine Learning
Data Science
ACTORS OF MACHINE LEARNING PROJECT
Banque de France, all rights reserved
Analyse the
data
Explore
the data
Trained model
Version
control
system
Model
deployment
Train the model,
optimise the
learning
Integration into
the software
• Security
• Unit test
• Process flow
• Logs
Monitoring :
• Performance
• Availability
• Quality
Maintain in
optimal condition
Data + Dev + Ops=> MLOps

16. Banque de France, all rights reserved
MLOPS
What is MLOps?

17. Banque de France, all rights reserved
1. Monitoring ML model in production is paramount
2. Version your Code, Model, Data
3. Test your data, test your model
4. Continuous delivery, continuous training
GOOD PRACTICES TO PUT YOUR ML MODEL IN PRODUCTION

18. Banque de France, all rights reserved
Q & A