Reproducible machine learning Steph Locke Reproducible if Data=Same + Analysis=Same Replicable if Data=Different + Analysis=Same Robust if Data=Same + Analysis=Different Generalisable if Data=Different + Analysis=Different It’s reproducible if… With the same ✨ environment With the raw ✨ data With the unmodified ✨ code = Produces exactly the same results Benefits for You/Team - Fewer headaches around environments and data - Less rework of code - Clear standards - Easier operationalisation Stakeholders - Stable results - Auditable - Maintainable / correctable - Easier operationalisation Recommendations FAIR data Findable - Unique name / ID - Documented Accessible - Common access methods - Open protocol - Metadata stored even after data may be removed Interoperable - Common standards - Terms defined Reusable - License and use rights specified - Provenance documented What to use Use - Logging - Version control - Fixed seeds* - Dependency tracking Framework examples - MLFlow - {drake} - Azure ML - FairML

1. Helping manufacturers innovate through AI
Reproducible machine learning
Steph Locke
@theStephLocke

2. Steph Locke
CEO @ Nightingale HQ
Data & AI specialist
Microsoft MVP, 5+ years
Microsoft MCT
T: @theStephLocke
Li: /stephanielocke
steph@nightingalehq.ai

3. Agenda

4. Reproducibility

5. Reproducibility
Data
Same Different
Analysis
Same Reproducible Replicable
Different Robust Generalisable

6. It’s reproducible if…
With the same ✨ environment
With the raw ✨ data
With the unmodified ✨ code
➔Produces exactly the same results

7. Benefits

8. You/Team
Fewer headaches around environments and data
Less rework of code
Clear standards
Easier operationalisation

9. Stakeholders
• Stable results
• Auditable
• Maintainable / correctable
• Easier operationalisation

10. Recommendations

11. Findable
- Unique name / ID
- Documented
Accessible
- Common access methods
- Open protocol
- Metadata stored even after data may
be removed
FAIR data
Interoperable
- Common standards
- Terms defined
Reusable
- License and use rights specified
- Provenance documented

12. Use
• Logging
• Version control
• Fixed seeds*
• Dependency tracking

13. Framework examples
• MLFlow
• {drake}
• Azure ML
• FairML

14. A trusted marketplace for
digital and AI products
and services for
manufacturers.
NightingaleHQ.ai
nightingale-hq
nightingalehqai
nightingalehq
Helping manufacturers
innovate through AI
14
Thanks!
• @theStephLocke
• steph@nightingalehq.ai