#IA Track - Technology and Tools Google Cloud Platform allows to build, to train and to serve ML models in serverless thanks to a dedicated service: AI-Platform. This service can do all, but, on the orediction serving part, there is others Google Cloud Platform services which can offer different characteristic and could be more interesting for some use cases - Big Query for batch prediction and based on Structured data - Cloud Run for online prediction, which brings Knative compatibility and thus portability through different operator and event on premise! During this talk, use cases, demos and implementation examples will be described based on Tensorflow already trained model. A comparaisons will be performed in term of deployment process, serving performances, team organisation and skills required, pricing and trade-off between portability and efficiency.

1. Which Severless platform
for serving a Tensorflow
model on Google Cloud
Guillaume Blaquiere
@gblaquiere
Lead developer/Scrum master Veolia

2. Which Severless platform
for serving a Tensorflow
model on Google Cloud
Guillaume Blaquiere
@gblaquiere
Lead developer/Scrum master Veolia

3. What is Serving in AI pipeline ?
Explore
data
Build
model
Train
model
Serve
model
Prepare
data

4. What is Tensorflow ?
Tensor Flow

5. What is Serverless ?
Operational
Model
Programming
Model
No Server
Management
Fully Managed
Security
Pay only for
usage
Service-based Event-driven Portable

6. AI platform on GCP

7. What is AI-Platform on GCP ?
Explore
data
Build
model
Train
model
Serve
model
Prepare
data
AI Platform
AI Platform
Cloud
Datalab
Cloud
Dataproc
Cloud
Dataprep
Cloud
Dataflow
BigQuery
Cloud Data
Fusion Datastudio AI platform

8. Serverless online prediction

9. AI-Platform serving features
G.A.
1 vCPU
2 Gb
No GPU
Beta
2 - 32 vCPU
2 - 208 Gb
1 - 8 GPU
$$

10. Portable solution
Any language
Any library
Any binary
Ecosystem of base images
Industry standard
.js .rb .go
.py .sh …
0 1 0
1 0 0
1 1 1
Containers

11. Containers
Flexibility
Serverless
Simplicity

12. Cloud Run
Container to
production in
seconds
Natively
Serverless
Run it anywhere
you want

13. Deploy in production in seconds
Container Registry
https://yourservice.run.app
Cloud Run

14. Cloud Run: Pay-per-use
CPU / Memory / Requests 100ms

15. Up to 80 concurrent requests
concurrency = 1
concurrency = 80

16. Cloud Run: Leverage concurrency

17. ● Common API and runtime environment for
serving workloads
● Implements learnings from Google and over 50
companies contributing
● Portability of experiences, tooling, and
workloads between Knative environments - you
can even run serverless on-prem
https://knative.dev
Portability based on Knative

18. Cloud Run
Fully managed, deploy your
workloads and don’t see the
cluster.
Cloud Run on Anthos
Deploy into Anthos, run
serverless side-by-side with
your existing workloads.
Knative Everywhere
Use the same APIs and
tooling anywhere you run
Kubernetes with Knative.
One experience, where you want it

19. - 1 vCPU, 2Gb of memory
- Scale to 0
- Pay-as-you-use
- Portable serverless container
Cloud Run summary

20. Serverless batch prediction

21. What is batch prediction process ?
Extract data
to files
Run batch
model
Store results
Load results in database
$ $ $ $
BigQuery Storage AI Platform Storage

22. BigQuery ML

23. SELECT data as input FROM `dataset.my_data`
SELECT * FROM ML.PREDICT(
MODEL `model.my_model`, (
Use model in a query
INSERT INTO result.my_results
)
)

24. BigQuery Storage AI Platform Storage
Batch prediction with BQML
$ $ $ $
Extract data
to files
Run batch
model
Store results
Load results in database

25. What to choose?

26. Online prediction
Sparse request
Spiky traffic
Compute intensive - GPU
Sustainable traffic
Cloud Run
AI-Platform
Portability
No IT engineering

27. Batch prediction
Structured data
Compute intensive - GPU
AI-Platform
Data reachable with BQ
(un)structured data
No data engineering
BigQuery
Not only Tensorflow model

28. Thank you!
AI-Platform cloud.google.com/ai-platform
BigQuery ML cloud.google.com/bigquery-ml
Cloud Run cloud.google.com/run
Knative knative.dev
Find me on :
Twitter @gblaquiere
GitHub guillaumeblaquiere
Medium @guillaume.blaquiere