Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

BigMLSchool: ML Platforms and AutoML in the Enterprise

An introductory session on the current situation of Machine Learning, the importance of ML platforms and AutoML, and why ML needs to be properly taught at schools and universities.

The lecturer is Ed Fernández, Board Director at BigML and Arowana International, a Private Equity firm, Faculty at Northeastern University (the Silicon Valley campus), lecturer at Headspring Corporate Learning (the Joint Venture of Financial Times and IE Business School), and mentor at Berkeley Sutardja Center for Entrepreneurship and Technology.

*Machine Learning School for Business Schools 2021: Virtual Conference.

  • Soyez le premier à commenter

BigMLSchool: ML Platforms and AutoML in the Enterprise

  1. 1. #BigMLSchool Agenda Machine Learning & AI* in Education: • Objectives & Challenges ML/AI* Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools Machine Learning & Education ML Platforms and AutoML
  2. 2. #BigMLSchool Agenda Machine Learning & AI* in Education: • Objectives & Challenges ML/AI* Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools *Disclaimer: The term AI* (Artificial Intelligence) refers specifically to  the ability to build machine learning driven applications which ultimately automate and/ or optimize business processes and SHOULD NOT BE CONFUSED with robust or strong Artificial Intelligence in the formal sense, ‘something not likely to happen for a least this decade and/maybe next’ (emphasis from the author)
  3. 3. Teaching & designing: Master’s Degree Courses Master in Enterprise Applied Intelligence & Master in Data Analytics • EAI6020 AI Systems & Technologies • ALY6080 Experiential Learning - Machine Learning & Analytics - Project Based • ALY6983 Special Topics: Applied Machine Learning Corporate Learning Machine Learning & AI - Enterprise Onboarding
  4. 4. #BigMLSchool About your instructor: • Nerd (Engineer in the 90s - 1st PC: Commodore 64, 64Kb RAM) • turned into Business (Corporate Executive) • turned Entrepreneur (still shareholder) • turned into VC (Startups, VC and PE) • turned into Board directorships (Non-Exec Board Director) • turned into Teaching (Northeastern University - Silicon Valley, Berkeley Center for Entrepreneurship & Technology, Headspring - IE Business School) Superskill: I can spin a [insert_object] in the air on the tip of a finger best way to learn anything, teach it! R. Feynman
  5. 5. #BigMLSchool ALY6080/6983: Experiential Learning - Machine Learning Motivation & Syllabus [6-12 weeks course] Focus on learning by doing, real life - sponsored Capstone project and theory, concepts and methods delivered with examples and use cases • Deep Learning vs Traditional ML • Supervised Learning I: creating a ML app end to end, Linear Regression - Decision Trees - Model Performance • Supervised Learning II: Logistic Regression, Random Forest & Ensembles, Bagging & Boosting, Neural Networks & DL • Unsupervised Learning: Clustering, Association Discovery, Anomaly Detection • Feature Engineering, Dimensionality Reduction - PCA and Automated ML • Deploying ML models - Capstone Project Tools & Technologies: Python, R (legacy) Tableau, PowerBI, BigML, AutoML (project/use case based) Pic credit: BigML AutoML platform https://github.com/whizzml/examples/tree/master/automl
  6. 6. #BigMLSchool EAI6020: AI Systems & Technologies Focus on Tools & Engineering for ML • Machine Learning & AI*-Industry Overview • ML/AI Engineering - Infrastructure & Tools (with Lab) • Data Engineering • Data Management • ML Deployment & Prediction Serving (with Use Case & Lab) • Data Architecture Evolution & Business Rationale (with Use Case) • Capstone Project (Use Case & Lab) Spark, SQL/NoSQL, Databricks, BigML References: • Full Stack Deep Learning course - Berkeley (2019-2020/ Sergey Karayev) • ML Systems Design - Stanford (2021/Chip Huyen) Motivation & Syllabus [12 weeks core course]
  7. 7. #BigMLSchool Objectives • Improve Practical Skills by exposure to Use Cases and Sponsored - company projects • Experiential Learning: Project Based, Practice first - Theory/Math later (learning by doing) • Improve Soft Skills: Communication, Synthesis & Objectivity, Team Collaboration, Customer Orientation, Sharing, working under pressure • Objective & Outcomes focus: engineering vs math, industry tools vs coding • Focus on learning by doing, real life - company sponsored Capstone project • Close Industry gap: get (many more) ML models to production
  8. 8. #BigMLSchool The Missing Course in Data Science The Missing Course in MBA MBA Data Science Technical Knowledge: Math Statistics Analytics ML Programming Python/R SQL/Databases Business Knowledge: Soft Skills Communication Teamwork & Collaboration Data Driven Decisions Digital Transformation Finance Leadership ML Engineering: Applied ML - Project Based ML MLOps Data Engineering Tools & Infrastructure Data Driven Leadership: ML Applications ML Project Management Data Science/ML team management Tools & Infrastructure Challenges: An Educational Gap MBAs &
  9. 9. #BigMLSchool Challenges Hands-on experience and practical application of ML is relegated in favor of theoretical and foundational knowledge (programming, math, statistics) - The opposite is also true • Select methods win over application oriented aspects • Power shift in Curriculum: Syllabus must meet students expectations e.g demand for advanced DL methods (GANs, Transformers), despite what ‘reality’ dictates (what you’ll need in a real job as Data Scientist or MBA/exec of Data Driven projects) • Students have (very) different backgrounds and levels of experience: Behavioral challenges due to diversity, cultural differences and diverging attitudes GOAL • Find optimal balance between teaching hands-on best practices, practical skills and technical skills/theory/concepts. Why do students need to spend 6-9 months learning to code before doing any ML?
  10. 10. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools
  11. 11. #BigMLSchool Jensen Huang, CEO of Nvidia
  12. 12. #BigMLSchool Machine Learning Adoption Toward mainstream source: courtesy of BigML Inc · http://bigml.com
  13. 13. #BigMLSchool Adoption Cycle: Machine Learning Platforms ML platforms: Custom Built vs Buy, crossing the chasm source: adapted from BigML Inc materials · http://bigml.com • Open Source • Custom Built vs Buy • Fragmented • Proprietary • Buy vs Build • Consolidated
  14. 14. #BigMLSchool credit: Full Stack Deep Learning Course - Infrastructure & Tools (*augmented with BigML & DataRobot Academic Programs) * link to free Academic Programs:
  15. 15. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools
  16. 16. #BigMLSchool Internal & External ML modeling, heuristics AI assets: ML platform AI assets: skills/expertise ML Adoption cross-function Enterprise Roadmap for AI & ML at scale
  17. 17. #BigMLSchool Scaling ML: dimensions Use Cases (sources of value) ML Models Generation ML Models in Production • Volume, both by the number of Models in Production and the ability to validate new experiments/ hypothesis quickly determine success • Significant number of models in production, complexity of ML workflows and model management call for tools & platform approach (ML platforms) • Rapid Model Prototyping driven by AutoML (Automated Machine Learning) for increased speed & efficiency Key activities • Experiments & rapid prototyping • Validation & testing • Model improvement/feature engineering • Model deployment • Performance measurement & monitoring • Model drift/Model lifecycle Management Key technologies /tools AutoML ML Platforms
  18. 18. #BigMLSchool How many ML models are too many models Facebook ML platform (a.k.a FBlearner): +1Mn ML models trained +6 Mn predictions/sec 25% of engineering team using it Source: ModelOps IBM research Waldemar Hummer et al
  19. 19. #BigMLSchool Architecture of a ML Platform ML at scale requires tooling and ultimately a platform approach ML Platform architecture - Courtesy of BigML
  20. 20. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Trends Conclusions & Recommendations • For Business Schools • For Technical Schools
  21. 21. #BigMLSchool Amazon Jeff Bezos’ letter to Amazon shareholders - May, 2017 “Machine learning and AI is a horizontal enabling layer. It will empower and improve every business, every government organization, every philanthropy — basically there’s no institution in the world that cannot be improved with machine learning” . Jeff Bezos
  22. 22. #BigMLSchool Machine Learning Platforms An Infrastructure & Service layer to drive ML at scale in the enterprise Facebook FBlearner May 9, 2016 https://code.fb.com/core-data/ introducing-fblearner-flow-facebook-s- ai-backbone/ Google TFX Tensorflow Aug 13, 2017 https://www.tensorflow.org/tfx/ https://dl.acm.org/ft_gateway.cfm? id=3098021&ftid=1899117&dwn=1&CF ID=81485403&CFTOKEN=79729647b 2ac491f-EAC34BCC-93F2-A3C5- BE9311C722468452 Netflix Notebook Data Platform Aug 16, 2018 https://medium.com/netflix-techblog/ notebook-innovation-591ee3221233 Uber Michelangelo Sept 5, 2017 https://eng.uber.com/michelangelo/ Twitter Cortex Sept, 2015 https://cortex.twitter.com/en.html https://blog.twitter.com/engineering/ en_us/topics/insights/2018/ml- workflows.html Magic Pony acquisition - 2016: https://www.bernardmarr.com/ default.asp?contentID=1373 AirBnB BigHead Feb, 2018 https://databricks.com/session/ bighead-airbnbs-end-to-end-machine- learning-platform LinkedIN Pro-ML Oct, 2018 https://engineering.linkedin.com/blog/ 2018/10/an-introduction-to-ai-at- linkedin
  23. 23. #BigMLSchool an unfair ‘platform’ advantage
  24. 24. #BigMLSchool Machine Learning Platforms eBay Krylov Dec 17, 2019 https://tech.ebayinc.com/engineering/ ebays-transformation-to-a-modern-ai- platform/ Lyft Flyte Jan 20, 2020 https://eng.lyft.com/introducing-flyte- cloud-native-machine-learning-and- data-processing-platform- fb2bb3046a59 AT&T Acumos Oct 30, 2017 https://medium.com/netflix-techblog/ notebook-innovation-591ee3221233 Spotify Spotify ML platform Dec 13, 2019 https://labs.spotify.com/2019/12/13/the- winding-road-to-better-machine- learning-infrastructure-through- tensorflow-extended-and-kubeflow/ Delta Airlines (licensed) Jan 8, 2020 https://www.aviationtoday.com/ 2020/01/08/delta-develops-ai-tool- address-weather-disruption-improve- flight-operations/ GE Predix (customer IoT platform) Feb, 2018 https://www.ge.com/digital/sites/ default/files/download_assets/Predix- The-Industrial-Internet-Platform- Brief.pdf KT Telecom Neuroflow Jan, 2018 https://disruptive.asia/kt-ai-platform- internal-use/ An Infrastructure & Service layer to drive ML at scale in the enterprise
  25. 25. #BigMLSchool 25 Increasing number of models & complexity Facebook Twitter Linkedin Google SO PUT THE RIGHT ML PLATFORM IN PLACE THESE COMPANIES ALREADY DID (Custom Built) •e-commerce •online/real time transaccions •consumer C2C services •Predictions driven by volume (millions) & models •long term trends & patterns •B2B & Government services •consumer C2C services •Predictions driven by certainty vs speed •rules based knowledge AirBnB Netflix Spotify GE AT&T Delta eBay Amazon Lyft Uber
  26. 26. MACHINE LEARNING AS A SERVICE MACHINE LEARNING PLATFORM & SOFTWARE https://www.crisp-research.com/vendor-universe/machine-learning/#fndtn-mlaas Machine Learning Platforms Vendor Landscape MLaaS: Machine Learning as a Service & On Premise Source:
  27. 27. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools
  28. 28. #BigMLSchool “All Models are wrong, but some are useful”
  29. 29. #BigMLSchool AutoML Typical AutoML pipeline AutoML Feature generation Feature selection Model selection = + + • Cluster Batch Centroids (Clustering) • Anomaly Scores (Anomaly Detection) • Batch Association Sets (Association Discovery): Using the objective field from your dataset as consequent and using leverage and lift as search_stra tegy • PCA Batch Projections (Principal Component Analysis) • Batch Topic Distributions (Topic Model): Created only when the dataset contains text fields. • Recursive Feature Elimination • automatically creating and evaluating multiple models with multiple configurations (decision trees, ensembles, logistic regressions, and deepnets) by using Bayesian parameter optimization. The OptiML algorithm is split into two phases. The first, the “parameter search” phase, uses a single holdout set to iteratively find promising sets of parameters. The second, the “validation” phase is used to iteratively perform Monte Carlo cross-validation on those parameters that are somewhat close to the best. References: • Introduction to Automatic Model Selection - OptiML https://blog.bigml.com/2018/05/08/introduction-to-optiml-automatic-model-optimization/ • Recursive Feature Elimination - Github https://github.com/whizzml/examples/tree/master/recursive-feature-elimination • Bayesian Parameter Optimization - Wikipedia https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization • Automated Machine Learning - OptiML: https://blog.bigml.com/2018/05/16/optiml-the-nitty-gritty/
  30. 30. #BigMLSchool AutoML Automated Machine Learning Problem Formulation Data Acquisition Feature Engineering Modeling and Evaluations Predictions Measure Results Data Transformations 5% 80% • Data acquisition and transformation - semi automated • Feature Engineering, key to model performance - semi automated 10% • Goal definition - Human driven 5% • Model Selection & Evaluation - automated • Measuring & Monitoring - automated
  31. 31. #BigMLSchool 31 Enable knowledge workers (e.g., analysts, developers) to build stable and insightful models quickly. Scale the number of predictive use cases in collaboration with non-technical peers through rapid prototyping. Best AutoML approaches rely on automation of parts of the Machine Learning process (e.g., hyper-parameter tuning) without limiting the practitioners’ ability to control customization. GDPR, data privacy, interpretability and prediction explanations become critical concerns when deploying AutoML AutoML Automated Machine Learning That feeling when your AutoML models are done
  32. 32. #BigMLSchool 32 AutoML DATAROBOT H2O BigML Data Preparation • Encoded categorical variables (one-hot); Text n- grams; Missing values imputing; Discretization (bins) • limited manual transformations • Max. of 10 classes in the objective* •Encoded categorical variables (one-hot); Missing values handling; Date-time fields expansion; Bulk interactions transformers; SVD numeric transformer; CV target encoding; Cluster distance transformer; Time lag •Automatic feature engineering possible when using AutoDL • Encoded categorical variables (one-hot); Text analysis; Missing values handling; Date-time fields expansion • Automatic Recursive Feature Selection & Feature Engineering • Multiple flexible manual transformations • Max of 1,000 classes in the objective Optimization Undisclosed optimization technique (“expert data scientists preset hyperparameter search space for models*) Random Stacking (a combination of random grid search and stacked ensembles, plus early stopping) Bayesian Parameter Optimization (SMAC — Sequential Model-based Algorithm Configuration) & DNN Metalearning Models/Algorithms •Open-source libraries: scikit-learn, R, H2O, Tensorflow (not CNN or RNN), Spark, XGBoost, DMTK, and Vowpal Wabbit 
 •They also “blend” multiple models during the optimization process. •GBMs, Random Forests, XGBoost, deep neural nets, and extreme random forests •· Stacks of models can be learned. Best of family stacks adopt the top model type from each of the main algorithms. •Decision trees, random decision forests, boosting, logistic regression, deep neural networks 
 •Customizable model ensembles with Fusions leveraging the individually optimized models for different classification, regression algorithms. Speed It tests 30-40 different modeling approaches and takes ~20 min. Default time limit for AutoML is 1 hour. Can use GPU or CPU. Can specify settings for accuracy, time, and interpretability. It tests 128 different modeling approaches (creating more than 500 resources) and takes ~30 min. Model Visualizations & Interpretability • Limited model visualizations • Feature importance for models • Predictions explainability • Dashboard: A single page with a global interpretable model explanations plot, a feature importance plot, a decision tree plot, and a partial dependence plot. • A machine learning interpretation tool (MLI) that includes a KLIME or LIME-SUP graph. • Multiple model visualizations to analyze the impact of the variables on predictions: sunburst, decision tree, partial dependence plots, line chart (LR) • Feature importance for models • Predictions explainability Model Evaluations • Confusion matrix
 • ROC curve (only for binary classification)
 • Lift curve (only for binary classification)
 • Side-by-side evaluations comparison
 • Trade-off between complexity vs. performance • Models are ranked by cross-validation 
 AUC by default. • Return leaderboard sortable by deviance (mean residual deviance), logloss, MSE, RMSE, MAE, RMSLE, mean per class error • Confusion matrix
 • ROC curve
 • Precision-Recall curve
 • Gain curve
 • Lift curve
 • Multiple evaluations comparison chart Programmability & Deployability • Models can be used and created via API • Export models
 • Cloud, VPC or on-premises • H2O allows you to convert the models you have built to either a Plain Old Java Object (POJO) or a Model ObJect, Optimized (MOJO). • H2O-generated MOJO and POJO models are ieasily embeddable in Java environments • Models can be used and created via API • Export models
 • Cloud, VPC or on-premises Source: Public Resources, Vendor Docs, BigML Analysis Metalearning!
  33. 33. #BigMLSchool 33 AutoML - Metalearning Automatic Network Hyperparameters Selection - DNNs (DeepNets) We trained 296,748 deep neural networks so you don’t have to! • 296,748+ deep neural networks trained on 50 datasets • For each one, recorded the optimum network structure for the given dataset structure (number of fields, types of fields, etc) • Trained a model to predict the optimum network structure for any given dataset. • This predicted network structure & hyper parameters can be used directly or as a seed for a more intensive network search Source: BigML - DeepNets https://blog.bigml.com/2017/10/04/deepnets-behind-the-scenes/ • Automated Machine Learning - OptiML: https://blog.bigml.com/2018/05/16/optiml-the-nitty-gritty/
  34. 34. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools
  35. 35. #BigMLSchool We are here (mostly) Simplified* AI Landscape * and imperfect Future: • Knowledge representation (symbolic/ Subsymbolic) • Planning (Reinforcement Learning, Agents) • Reasoning (Causality, Logic, Symbolic) • Search & Optimization (evolutionary/ genetic algos)
  36. 36. #BigMLSchool 36 BigML, Inc Private and Confidential BigML Product Progression 5 AutoML, Linear Regression, Node- Red, Workflow Report, Improved Topic Modeling Organizations, Operating Thresholds, OptiML, Fusions, Data Transformations, PCA Boosted Trees, ROC Analysis, Time Series, DeepNets Scripts, Libraries, Executions, WhizzML, Logistic Regression, Topic Models Association Discovery, Correlations, Samples, Statistical Tests Anomaly Detection, Clusters, Flatline Evaluations, Batch Predictions, Ensembles, Starbursts Core ML Workflow: Source, Dataset, Model, Prediction Prototyping and Beta 2019 2018 2017 2016 2015 2014 2013 2012 2011 Automating Model Creation, Selection, Operation and Workflows = Making Machine Learning Easier Reproducibility at the core: Programmability, Interpretability, Explainability are essential part of BigML's platform Sophistication Ease of Use WE HAVE BEEN BUILDING A STRONG FOUNDATION TO DEVELOP, DEPLOY AND OPERATE MACHINE-LEARNING BASED APPLICATIONS OF UNPARALLELED QUALITY
  37. 37. #BigMLSchool 37 BigML, Inc Private and Confidential 7 AI/ML Market Maturity Automating Workflows for Model Creation, Selection, Operation Extending the Platform to Build and Manage Smarter Predictive Applications End-to-End Building the BEST End- to-End Machine Learning Platform 2020 2030 1980 BigML's Co-Founder Participates in first University Machine Learning 2011 BigML Founded BigML Future EXTENDING THE PLATFORM TO BUILD AND MANAGE SMARTER PREDICTIVE APPLICATIONS END-TO-END Reasoning Knowledge Representation Planning Optimization Principles Machine Learning ROBUST AI Doing to Reasoning, Planning, Knowledge Representation and Optimization what we have done to Machine Learning and combining them to build Robust AI Applications Machine Learning
  38. 38. #BigMLSchool next Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools
  39. 39. #BigMLSchool Recommendations Context The world is changing…. fast: • Companies are adopting ML/AI* quickly, the build- vs-buy paradigm is changing • ML tools & platforms are spreading (buy vs build/open source) • Success out there is measured by the ability to deploy models rapidly and efficiently • Technical debt in ML is an issue, MLOps and Engineering becoming critical (time to model deployment => time to market) • Students, technical or not, will confront a world where they’ll be expected to understand and (somehow) master ML/AI end to end • Tools & Platforms are here to help, coding necessary but not core to problem and solution (code automation and scripting)
  40. 40. #BigMLSchool Recommendations II For Educators, Business Schools and Technical Schools • MBAs and business leaders need to understand tech/ML/AI • Include ML in the Curriculum, industry approach, key concepts and high level ML modeling (no hard coding but use of scripting tools) • Experiential learning, hands on project assignments, tie ML models and use cases to business value. • Technical students need more Soft skills (comms, teamwork, project management). • MBAs need more ‘hard’ tech skills (tools, applications & tech concepts) AI
  41. 41. #BigMLSchool End Machine Learning & AI* in Education: • Objectives & Challenges ML/AI Industry Status: • ML Adoption • Scaling ML in the Enterprise ML Platformization AutoML • Future Evolution Conclusions & Recommendations • For Business Schools • For Technical Schools

×