This session provides a look at where and how Deltatre, a global leader for sport business services, is early-adopting AI techs in their world-class, multi-device OTT (over-the-top) platform services, powering some of the major sport federations, such as NFL Game Pass, ATP Tennis TV and FINAtv. Based on Azure AI services, 3rd party providers and custom models built with open-source libraries and frameworks, the session shows how AI is helping to achieve better system monitoring, improved customer support, more fan engagement and per-user customization in on-demand and live streaming.

1. Where and how can AI be used in a real-world
multimedia platform for sport?
Gianni Rosa Gallina
Milan | November 29 - 30, 2018

2. 2
This work is licensed under a Creative Commons License
All product and company names are trademarks™ or registered® trademarks of their
respective holders. Use of them does not imply any affiliation with or endorsement by them.

3. 3
OTT (over-the-top) is a term used in
broadcasting and technology business
reporting to refer to audio, video, and
other media transmitted via the
Internet as a standalone product
What is an OTT platform?

4. 3
IT’S NOT A (simple) VIDEO PLAYER
What is an OTT platform?

5. 4
SUBSCRIPTION
& BILLING
MANAGEMENT
REPORTING
MARKETING
& SOCIAL MEDIA
CUSTOMER
SUPPORT
TECHNICAL
SUPPORT &
OPERATIONS
EDITORIAL
TECHNICAL
PLATFORMS

6. 5
High quality, low-latency video streaming (live/VoD)
Global scale (potentially)
Very narrow load/usage peaks
Fine-grained content rights management
Sport-specific needs

7. 6
Deltatre’s OTT platform for sport events
Cloud based + hybrid scenario supported
fully hosted in Microsoft Azure as PaaS
11 end-user platforms
Full coverage
Unified

8. 7
Architecture
Courtesy of Alberto Chiavazza
Deltatre Solution Architect

9. 8
Architecture
Data
lake
OTT Platform API
Apps
API
Forge CMS
Diva
BackOffice
End-user
Apps
Diva
Player
Third Parties
payments, marketing, entitlement, support
Azure IaaS & PaaS
Customer’s
platform / data
ingestion

11. Third Parties
payments, marketing, entitlement, support
10
Where can we use AI?
Data
lake
Forge CMS
Diva
BackOffice
End-user
Apps
Diva
Player
Azure IaaS & PaaS
Customer Support,
Analytics & Monitoring
Platform
Monitoring
Content
Editing
Video
Indexing
Recommendation
Next Gen Content
& Customization
Patterns,
Behaviors &
Analytics

12. 11
AI for Customer Support
Zendesk Satisfaction Prediction
Predict bad customer satisfaction ratings before they happen
https://www.zendesk.com/blog/science-behind-satisfaction-prediction/
Zendesk Content Cues
Automatically reviews incoming Zendesk Support tickets: recognizes &
classify content in the tickets, provides suggestions for relevant
Knowledge Base help articles that should be created or updated
https://www.zendesk.com/blog/announcing-content-cues/

13. 12
AI for Analytics & Monitoring
Conviva Video AI Platform
Detects and diagnoses issues that impact consumer experience and
engagement. Find and analyze relationships to very quickly track
down exactly what entity in the end-to-end streaming hierarchy might
be responsible for problems in service delivery. It can also be used for
making content recommendations or understanding device usage by
application, channel, or show type.
https://www.conviva.com/video-ai/

14. 13
AI for Platform Monitoring
Application Insights – Smart Detections
Automatically warns of potential performance problems in web
applications. It performs proactive analysis of the telemetry that apps
send to Application Insights. If there is a sudden rise in failure rates, or
abnormal patterns in client or server performance, an alert is sent.
https://docs.microsoft.com/en-us/azure/application-insights/app-insights-proactive-diagnostics

15. 14

16. 15
AI for Content Editing
Smart Editor
Helps writing outstanding articles
to increase quality, virality and
sport fans engagement.
Automatic text analysis to provide
content auto-tagging and entities
identification (people, places,
etc.). Suggest additional content,
images or videos to reference,
from internal CMS content and/or
web.

17. 16
Demo

18. 17
Behind the scenes
Microsoft Cognitive Services
Text Analytics API, Bing Search API
https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics/
https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/
https://azure.microsoft.com/en-us/services/cognitive-services/bing-entity-search-api/
Free, open source WYSIWYG editor built for the modern web
https://quilljs.com/
https://github.com/quilljs/quill/

19. 18
Demo Architecture
Search API
Smart Editor
API
Frontend
Backend
Microsoft
Text Analytics
API
Microsoft
Bing Search API
Quill
Editor
Tags
Related content
ASP.NET Core
ASP.NET Core
ASP.NET Core / SPA

20. 19
Microsoft Cognitive Services
Knowledge
Map complex
information and
data in order to
solve tasks such as
intelligent
recommendations
and semantic
search.
Language
Allow your apps to
process natural
language with pre-
built scripts,
evaluate sentiment
and learn how to
recognize what
users want.
Search
Add Bing Search
APIs to your apps
and harness the
ability to comb
billions of
webpages, images,
videos, and news
with a single API
call.
Vision
Image-processing
algorithms to
smartly identify,
caption and
moderate your
pictures.
Speech
Convert spoken audio
into text, use voice for
verification, or add
speaker recognition to
your app.
https://azure.microsoft.com/en-us/services/cognitive-services/

21. 20
Language
Text Analytics API
Bing Spell Check API Web Language Model API Linguistic Analysis API
Language Understanding
Intelligent Service
Translator

22. 21
Why use them?

23. 22
AI for Video Indexing
Video Content Indexer
Makes video content actionable
(search, discover, recommend).
Audio transcription and speaker
recognition, auto-captioning, shot
change detection, content auto-
tagging (objects, places), text
OCR, famous / custom people
face identification, emotion
understanding

24. 23
Behind the scenes
Microsoft Cognitive Services
Computer Vision API, Face API
https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/
https://azure.microsoft.com/en-us/services/cognitive-services/face/
Microsoft Video Indexer
https://vi.microsoft.com/
Microsoft Azure Functions
https://azure.microsoft.com/en-us/services/functions/

25. 24
Demo Architecture
Microsoft
Video Indexer
Portal
NewFile
Handler
VideoAnalyzer
Frontend
Azure Function
Microsoft
Video Indexer
API
HTTP Req
MP4
Video
Blob Storage
Trigger
VideoIndexer
Complete
Callback
HTTP Req
VideoIndexer
Complete
QueueHandler
Blob Storage
Annotations
Job Status
Azure Function
Azure FunctionAzure Function

26. 25
Demo

27. 26
Microsoft Cognitive Services
Knowledge
Map complex
information and
data in order to
solve tasks such as
intelligent
recommendations
and semantic
search.
Language
Allow your apps to
process natural
language with pre-
built scripts,
evaluate sentiment
and learn how to
recognize what
users want.
Search
Add Bing Search
APIs to your apps
and harness the
ability to comb
billions of
webpages, images,
videos, and news
with a single API
call.
Vision
Image-processing
algorithms to
smartly identify,
caption and
moderate your
pictures.
Speech
Convert spoken audio
into text, use voice for
verification, or add
speaker recognition to
your app.
https://azure.microsoft.com/en-us/services/cognitive-services/

28. 27
Vision

29. 28
Computer Vision API
Image analysis
Content understanding
OCR
Text recognition (where and what)
Thumbnail generation
Smart scaling/cropping
Celebrities and Places
Recognize more than 200,000 celebrities from
business, politics, sports and entertainment, as
well as 9,000 natural and manmade landmarks
from around the world

30. 29
Computer Vision API
Image type
Clip Art Type 0 Non-clipart
Line Drawing Type 0 Non-Line Drawing
Black & White Image False
Content
Categories [{ “name”: “people_swimming”, “score”: 0.099609375 }]
Adult Content False
Adult Score 0.18533889949321747
Faces [{ “age”: 27, “gender”: “Male”, “faceRectangle”:
{“left”: 472, “top”: 258, “width”: 199, “height”: 199}}]
Main colors
Dominant Color Background White
Dominant Color Foreground Grey
Dominant Colors White

31. 30
OCR
JSON:
{
"language": "en",
"orientation": "Up",
"regions": [
{
"boundingBox": "41,77,918,440",
"lines": [
{
"boundingBox": "41,77,723,89",
"words": [
{
"boundingBox": "41,102,225,64",
"text": "LIFE"
},
{
"boundingBox": "356,89,94,62",
"text": "IS"
},
{
"boundingBox": "539,77,225,64",
"text": "LIKE"
}
...

32. 31
Emotion API
Face recognition
"faceRectangle": {"width": 193,
"height": 193,
"left": 326,
"top": 204} …
Emotion scores
"scores": { "anger": 5.182241e-8,
"contempt": 0.0000242813,
"disgust": 5.621025e-7,
"fear": 0.00115027453,
"happiness": 1.06114619e-8,
"neutral": 0.003540177,
"sadness": 9.30888746e-7,
"surprise": 0.9952837}

33. 32
Face API
Face detection
Detect one or more human faces in an image
Face verification
Check the likelihood that two faces belong to the same
person
Similar face search
Easily find similar-looking faces
Face grouping
Organize many unidentified faces together into groups,
based on their visual similarity
Face Identification
Search, identify, and match faces in your private repository
of up to 1 million people

34. 33
Video Indexer
Upload your video, and go!
Just upload your video, and start finding insights right
away, without writing a single line of code.
Make your content more discoverable
Quickly and easily extract spoken words, faces,
characters, and emotion.
Improve engagement with your videos
Metadata extracted by Video Indexer can be used to
build powerful engagement experiences with
recommendations, highlight clips, and interactive
videos.

35. 34
AI for Recommendation
Recommendation Engine
Suggests users which videos
to watch, based on theirs (and
others) watching history.
Custom ML model based on
Collaborative Filtering that
match content to users,
according to other users’
similar behavior (latent
factors).

36. 35
Demo

37. 36
Behind the scenes
Open-source library that simplifies training fast and accurate
neural nets using modern best practices. Includes "out of the
box" support for vision, text, tabular and collaborative filtering
models.
https://www.fast.ai/
https://github.com/fastai/fastai
Python-first, open-source deep learning framework
https://pytorch.org/
https://github.com/pytorch/pytorch
Python REST API (Hug)
http://www.hug.rest/

38. 37
Demo Architecture
Recommendation
API
Frontend
Backend
ASP.NET CoreCLI API Test Client Python Web App
HUG
API
Fast.AI
PyTorch
ML Model

39. 38
AI for Next Gen Content
Immersive 3D scenes from 2D videos
PoC prototype «Soccer On Your Tabletop»
from University of Washington, Facebook & Google Researchers
https://grail.cs.washington.edu/projects/soccer/
A system that transforms a monocular video of a soccer game into a
moving 3D reconstruction, in which the players and field can be
rendered interactively with a 3D viewer or through a Mixed/Augmented
Reality device.

40. 39
Demo

41. 40
Q&A

42. 41
Thank you!

43. 42
Ing. Gianni ROSA GALLINA
R&D Specialist, Senior Software Engineer @ Deltatre
▪ Virtual/Augmented/Mixed Reality
▪ AI, Machine Learning
▪ Immersive video streaming & 3D graphics for sport events
▪ Cloud solutions, web backends, serverless, video workflows
▪ Mobile apps dev (Windows / Android / Xamarin)
▪ End-to-end solutions with Microsoft Azure
gianni@rosagallina.com
@giannirg
http://gianni.rosagallina.com
Author