More Related Content Similar to Building Notebook-based AI Pipelines with Elyra and Kubeflow (20) More from Databricks (20) Building Notebook-based AI Pipelines with Elyra and Kubeflow2. About
DEG / Nov 18, 2020 / © 2020 IBM Corporation
– @MLnick on Twitter, Github, LinkedIn
– Principal Engineer, IBM CODAIT (Center for
Open-Source Data & AI Technologies)
– Machine Learning & AI
– Apache Spark committer & PMC
– Author of Machine Learning with Spark
– Various conferences & meetups
2
3. Improving the Enterprise AI Lifecycle in Open Source
DEG / Nov 18, 2020 / © 2020 IBM Corporation 3
– CODAIT aims to make AI solutions
dramatically easier to create,
deploy, and manage in the
enterprise.
– We contribute to and advocate for
the open-source technologies that
are foundational to IBM’s AI
offerings.
– 30+ open-source developers!
Center for Open Source Data & AI Technologies
codait.org
CODAIT
Open Source @ IBM
6. Workflow spans teams …
6
Data Analyze Process Train Deploy
Predict
&
Maintain
DEG / Nov 18, 2020 / © 2020 IBM Corporation
Data Engineers Data Scientists & Researchers
Machine Learning &
Production Engineers
7. … and tools
7
Data Analyze Process Train Deploy
DEG / Nov 18, 2020 / © 2020 IBM Corporation
Data formats
• CSV, SQL
• JSON,
Parquet,
AVRO
• Binary
(image,
audio)
• …
Data Engineers Data Scientists & Researchers
Machine Learning &
Production Engineers
Analysis & data
viz
• ggplot
• dplyr
• matplotlib
• Pandas
• SparkSQL
• …
Pre-processing
& pipelines
• dplyr
• pandas
• scikit-learn
• SparkSQL /
SparkML
• …
Frameworks
• R, scikit-
learn
• SparkML
• TensorFlow
• PyTorch
• LightGBM,
XGBoost
• …
Formats &
mechanisms
• Variety of
formats
• Containers
• …
10. Interactive Notebooks
DEG / Nov 18, 2020 / © 2020 IBM Corporation 10
Notebooks have become
the de-facto standard
for content-rich,
interactive & iterative
work
* Logos trademarks of their respective projects
11. Elyra Overview
DEG / Nov 18, 2020 / © 2020 IBM Corporation 11
Elyra is a set of AI-
centric extensions to
JupyterLab Notebooks
* Logos trademarks of their respective projects
12. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 12
– Visual Pipeline Editor
Visual editor for building AI pipelines,
enabling the conversion of multiple
notebooks into batch jobs or workflows.
– Notebooks as batch jobs
– Python script execution
– Automated Table of Contents
– Code Snippets
– Git integration
13. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 13
– Visual Pipeline Editor
– Notebooks as batch jobs
Extends the notebook UI to simplify the
submission of notebooks as a batch job
for model training
– Python script execution
– Automated Table of Contents
– Code Snippets
– Git integration
14. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 14
– Visual Pipeline Editor
– Notebooks as batch jobs
– Python script execution
Edit and execute python scripts against
local or cloud-based resources
– Automated Table of Contents
– Code Snippets
– Git integration
15. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 15
– Visual Pipeline Editor
– Notebooks as batch jobs
– Python script execution
– Automated Table of Contents
Generate & navigate table of contents
from notebooks & python scripts
– Code Snippets
– Git integration
16. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 16
– Visual Pipeline Editor
– Notebooks as batch jobs
– Python script execution
– Automated Table of Contents
– Code Snippets
Easy creation and insertion of reusable
code snippets for various languages
– Git integration
17. Elyra Key Features
DEG / Nov 18, 2020 / © 2020 IBM Corporation 17
– Visual Pipeline Editor
– Notebooks as batch jobs
– Python script execution
– Automated Table of Contents
– Code Snippets
– Git integration
Track project changes and share among
teammates
18. DEG / Nov 18, 2020 / © 2020 IBM Corporation
Getting started with Elyra
1. Try Elyra from Binder
ibm.biz/elyra-demo
2. Run Elyra from Docker
ibm.biz/elyra-docker-installation
3. Install Elyra on your local machine
ibm.biz/elyra-installation
18
19. DEG / Nov 18, 2020 / © 2020 IBM Corporation 19
20. Start using Elyra today!
Getting started with Elyra
ibm.biz/elyra-installation
Elyra on Github
github.com/elyra-ai/elyra
Elyra Notebook projects on Github
github.com/CODAIT/flight-delay-notebooks
github.com/CODAIT/covid-notebooks
Contributing to the projects
• Star and fork, submit bug reports, suggest improvements,
help with code reviews, join our community meetings
ibm.biz/elyra-demo
gitter.im/elyra-ai/community
DEG / Nov 18, 2020 / © 2020 IBM Corporation 20
22. DEG / Nov 18, 2020 / © 2020 IBM Corporation 22