Operationalizing Machine Learning

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rajeev Dutt
DimensionalMechanics Inc.
Operationalizing Machine Learning

2. Challenge: ML is not IT Ready

3. Data
Provenance
Scope
Quality Versioning
Integration Deployment
Discoverability Security
Monitoring Support
Lifetime
Key considerations for enterprise AI:

4. For AI to become ubiquitous IT has to be
able to manage AI model development
and deployment process!

5. Copyright DimensionalMechanics 20185
NeoPulse™ automates the creation, distribution and management of AI
Humans don’t have to create AI Models – AI Studio can do this for you

6. Ubiquitous (adjective): present, appearing, or
found everywhere.
NeoPulse™ is about making AI ubiquitous –
on every device, in the cloud, and on premise
for every business large and small for any
machine learning problem.
NeoPulse™ reduces the barrier to entry so
that any developer, regardless of machine
learning experience, can create, deploy and
manage custom AI models in one third the
time and at 10% of the cost of comparable
platforms.

7. Data
Provenance
Scope
Quality Versioning
Integration Deployment
Discoverability Security
Monitoring Support
Lifetime
NeoPulse® can help with

8. Introducing NeoPulse™:
One platform for all Enterprise AI needs
Integrate
Plan
Deploy
Design
Train
Analyze
Manage

9. Portable Inference Models (PIM)
A neural network that is encapsulated in a container that can be queried using a runtime layer,
also referred to as an AI Model
NeoPulse® AI Studio: AI to build AI
Server application with a powerful AI called “the oracle” that is capable of automating the
process of creating sophisticated AI Models
NeoPulse™ Query Runtime
A program that is licensed by the organization to allow any application in the enterprise to
access the AI model using a web-based (REST) API
NeoPulse™ Modeling Language (NML)
An intuitive DSL (domain specific language) developed by DimensionalMechanics™ that is
executed by the NeoPulse™ AI Studio to automate the creation of new AI Models
NeoPulse® Framework

10. Learning methods
Classification
Regression
Some unsupervised (eg. Auto-encoders, GAN)
Data types supported
 Audio
 Video – single frame
 Video – multiple frames (motion video)
 Images
 Text
 Numerical
 Time Series
 Medical data (DICOM)
Enterprise Features
 Automated AI engineering
 Multi-platform (Cloud, PCs, ARM64 devices)
 RESTful Interfaces
 Extensive logging
 Integration with enterprise workflows
 Enterprise scaling
 Nvidia CUDA GPU computing
Capabilities

11. PIM
NeoPulse® AI Studio NeoPulse® Query
Runtime
NML File
.CSV
NeoPulse® Workflow
REST
Application

12. PIMNeoPulse™ AI Studio
NML File
.CSV
With a simple
command, the PIM can
be deployed anywhere
there is a runtime
including on ARM64
devices
NeoPulse® Workflow

13. Create intelligent applications in 7 steps

14. With tumors
Without tumors
0
1
label path
0 /images/negative/img_n_0001.jpg
0 /images/negative/img_n_0002.jpg
0 /images/negative/img_n_0003.jpg
…
0 /images/negative/img_n_<…>.jpg
1 /images/positive/img_p_0001.jpg
1 /images/positive/img_p_0002.jpg
1 /images/positive/img_p_0003.jpg
…
1 /images/positive/img_p_<…>.jpg
Curate your data and construct a CSV file
1
lung.csv
Assuming that you have high quality images and properly formatted, a simple script can construct the csv
file – less than an hour

15. Create the NML script
2
lung_classify.nml
Copy one of the examples listed on the DM Github page and modify it for your needs – less
than an hour
oracle("mode") = "classification"
source:
bind = "/DM-Dash/medical/lungtumor/lung.csv" ;
input: x ~ from “path" -> image: [shape=[28, 28], channels=1] -> ImageDataGenerator: [rescale= 0.003921568627451];
output: y ~ from “label”-> flat: [2] -> FlatDataGenerator: [] ;
params: batch_size=32, number_validation=10000 ;
architecture:
input: x ~ image: [shape=[28, 28], channels=1] ;
output: y ~ flat: [2] ;
x -> auto -> y ;
train:
compile: optimizer = auto, loss = auto, metrics = ['accuracy'] ;
run: epochs = 4 ;
dashboard: ;
1
2
3
4
5
6
7
8
9
10
11
12
13
14

16. Compile and start training
3
Compiling the NML code (assuming no syntax errors) is immediate – seconds
…training is another matter
NeoPulse™ AI Studio
lung_classify.nml
lung.csv

17. Training…
4
Training can take time depending on the volume of data and the compute resources available. There’s nothing for
you to do but the machine will be busy for a couple of days or more for a decent model
Fortunately AI Studio employs a queuing model – so it doesn’t stop you from starting the next project
NeoPulse™ AI Studio

18. Export a PIM file
NeoPulse™ AI Studio
Exporting a PIM file is simple – choose from a set of models based on accuracy (for example) and simply
export in a single call. The PIM is a file that can be moved from one machine to another (either locally or in
the cloud)
lung_tumor_model.pim
5

19. Import a PIM file
6
NeoPulse™ Query Runtime
lung_tumor_model.pim
Once the model has been built, you can move the resulting PIM file from machine to machine as long as the
NeoPulse Query Runtime has been installed. Importing the model into the runtime is a simple command –
takes just a couple of seconds.

20. Call the model via a REST API from an application
7
NeoPulse™ Query Runtime
After importing the model, NeoPulse™ Query Runtime automatically generates a RESTful API that allows
applications to query the model directly. You don’t need to build any custom APIs to call your model.

21. NeoPulse® Advantages
50x less code
18x lower project cost
4x lower developer cost
3x shorter project duration 3 4
18
50
1 1 1 1
0
10
20
30
40
50
60
Project Duration Development
Cost
Project Cost Lines of Code
Competition DM

22. CASE STUDY: AMORE PACIFIC
Sept 2015: Amore Pacific added to Forbes “Most Innovative
Company” list. Revenue: $4.5B
May 2018: Amore Pacific sent a team of four engineers without
ML experience for a 1 month training session on NeoPulse®
June 2018: Four engineers develop 20 models in 20 days during
initial training. Return to South Korea with 100%
recommending DM as company platform for AI model
development.
Sept 2018: Company hires Chief Digital Technology Officer to
oversee AI team.
 Amore Pacific puts first DNN Model into production with
$1200/month revenue stream to DM.
DimensionalMechanics has created a platform called
NeoPulse that maes it possible for companies like AMORE
PACIFIC to do machine learning at scale. We also found the
training useful and comprehensive. By using the NeoPulse, I
am confident that it will be a great help in achieving our # 1
beauty AI company vision."
-Kevin Choi, Leader, Digital IT Innovation Team, Amore
Pacific

23. Case Studies
A Seattle based staffing startup was quoted nearly $450,000
to develop a solution for their AI platform – NeoPulse
developed the solution for under $10,000.
A Seattle based medical startup was paying $20,000/month
for 6 months to develop and maintain a solution to create an
AI model that was 74% accurate. In three days, NeoPulse
built a solution with 86% accuracy costing the company only
$4,000.
Stanford University physicians could create a model to
differentiate between normal and abnormal PET/CT images
on NeoPulse with no prior knowledge of AI. Quality was high
enough that they published the results and the paper got
accepted at RSNA.
University of Washington ran a study comparing NeoPulse to
a standard vision recognition algorithm called VGG16. It took
20,000 iterations using VGG16 to get to 95% accuracy. It took
NeoPulse 30 iterations to reach the same accuracy.

24. Material Science
Auto-generation of materials:
• With University of Washington, we are running a program to automatically understand
material properties and then generate new materials with those properties
• Imagine designing new ultra-strong materials automatically and then 3D printing them…
AI generated materials
Original material

25. Available as AMIs on AWS AI Marketplace

26. NeoPulse® AI Studio

27. NeoPulse® Query Runtime

28. Watch out for a major announcement at
AWS re:Invent

29. Thank you

30. NeoPulse® AMI Architecture
Optional

31. Enterprise ML Pitfalls
Model Quality
How accurate is our model?
What is the false positive/false negative rate? RoC curve?
Is my model overfitted?
Bias vs. Variance of model?

32. Enterprise ML Pitfalls
Version Control
What version of the model am I working with?
When was it created?
Can I roll back to a previous model if the current model does not
perform?

33. Enterprise ML Pitfalls
Integration
Can the process of creating and deploying AI models be integrated
in an enterprise workflow?
How easy is it to integrate the models into enterprise
applications?
Are there any standards?

34. Enterprise ML Pitfalls
Deployment
Where has my model been deployed?
How many versions exist?
Who determines when and how the model is deployed?
What is the target environment (OS, Memory, Hardware config.)?

35. Enterprise ML Pitfalls
Discoverability
What models exist?
How do I access them?
Where do the models exist?

36. Enterprise ML Pitfalls
Monitoring
Can I retrieve statistics about my model?
 Number of queries
 Errors
 Time taken per query
 Batch vs. real time
 Overall performance metrics: CPU and memory utilization

37. Enterprise ML Pitfalls
Data Provenance
Where did the data to train the model come from?
How trustworthy is the data?
Is it biased?

38. Enterprise ML Pitfalls
Scope
What does my model do? (ex. classification of dogs/cats/lemurs)
What are its limits? – what can it do what can’t it do?
Who is the audience of the model?
What technology is used by the model?
Where can it run?
Who created it? For what purpose?

39. Enterprise ML Pitfalls
Security
Has anyone tampered with the training data? Can I tell?
How do I know that I can trust the model and that no one has
tampered with it?
Access controls on the model?
Is it possible to reverse engineer the model or the training data?*

40. Enterprise ML Pitfalls
Support
Do I have the staff with the right skills to support the solution?
What kind of problems am I likely to see?
How do I validate/test the model in the wild?

41. Enterprise ML Pitfalls
Lifetime & Deprecation
Does the underlying training data change?
How often?
Are the fundamental model statistics changing over time?
Should I retrain?