OSGi Community Event 2018 Presentation by Tim Verbelen (imec) Abstract: Nowadays AI is reaching new heights on the hype cycle, especially due to recent advances in deep learning techniques. A lot of deep learning frameworks exist for creating and training deep neural networks, the most popular ones being PyTorch and TensorFlow. However, how to integrate, deploy and manage these neural networks in complex software systems is often overlooked. In this talk we show how OSGi can be used as a modular runtime for deep learning models. We embed those models inside OSGi bundles, and use the extender pattern to make these available as OSGi services. You can then use your favorite OSGi specs such as DS and PushStreams to integrate these into your application.

1. PUBLIC1
OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME
TIM VERBELEN
Senior Researcher
imec – Ghent University

2. PUBLIC
WHAT IS A.I.?
2
THE NEXT BIG THING (AFTER BLOCKCHAIN OFC!)

3. PUBLIC
WHAT IS MACHINE LEARNING?
3
LEARNING A FUNCTION APPROXIMATION MAPPING INPUTS TO OUTPUTS
ModelModel outputsinputs

4. PUBLIC
WHAT IS DEEP LEARNING?
4
USING DEEP NEURAL NETWORKS AS MACHINE LEARNING MODEL
Neural NetworkNeural Network
“cat”

5. PUBLIC
DEEP LEARNING FRAMEWORKS AND TOOLS
5

6. PUBLIC
THE AVERAGE DATA SCIENTIST WORKFLOW
1. Inspect and clean up the data
2. Select and encode features / outputs
3. Script together a model training procedure
4. Find some good hyperparameters
5. Dump the trained model
6

7. PUBLIC
BUT WHAT ABOUT PRODUCTION?
7

8. PUBLIC
TENSORFLOW SERVING
8
Client
Servable
Handler
Dynamic
Manager
VersionPolicy
models/
1/
…
2/
…
Server
gRPC/REST
request

9. PUBLIC
CHECK THE BOXES
I want to query my models with an RPC/REST call
I train my models using TensorFlow
I deploy my models on a containerized infrastructure
I don’t need additional metadata/versioning besides a single
incrementing integer
9

10. PUBLIC
OSGI CAN HELP!
 A unit of deployment
 Package ML model as an OSGi bundle
 A resolveable artifact with requirements and capabilities
 Requirements: what do you need to run this ML model
 Capabilities: what kind of inputs can it process and what kind of outputs
does it give you?
 A lightweight service model
 Access your model via an OSGi service
 Service selection at runtime
 Use service properties and target filters to select the best model at runtime
10

11. PUBLIC
A WAY TO SHARE ML MODELS
A common format for describing computation graphs
 defined as protocol buffers
 specifies data types and operators
 framework agnostic
11
OPEN NEURAL NETWORK EXCHANGE (ONNX)

12. PUBLIC
A WAY TO EXECUTE MODELS IN OSGI
 A modular deep learning framework developed in OSGi
 Neural networks, operators, datasets, learners, … as an OSGi service
 Low-level operations via JNI and blas/cublas/cudnn backends
 Distributed deployments using OSGi remote services
 Web UI to build, deploy, train models
12
DIANNE FRAMEWORK
http://dianne.intec.ugent.be

13. PUBLIC
DEMO TIME
13

14. PUBLIC
NOW LET’S ADD A ROBOT
14

15. PUBLIC
NOW LET’S ADD A ROBOT
15

16. PUBLIC
PUTTING THINGS TOGETHER
16
@Reference Camera camera;
@Reference NeuralNetwork nn;
@Reference ArmController controller;
public Promise<Boolean> grasp(){
return camera.stream() // returns PushStream<Frame>
// timeout after 10 seconds
.timeout(Duration.ofSeconds(10))
// process camera frame with neural network
.map(frame -> nn.forward(toTensor(frame))
// use result to update controller
.map(r -> controller.update(r))
// short circuit in case of success
.anyMatch(s -> s);
}

17. PUBLIC
NOW LET’S ADD A ROBOT
17

18. PUBLIC18

19. PUBLIC19
THANK YOU!
http://dianne.intec.ugent.be