Session from the North American Collaboration Summit on using Azure Cognitive Services to check images taken from a Raspberry Pi for the presence of our food truck. If present it informs the user in MS Teams or Skype.

1. OFFICE 365, AZURE AND A
RASPBERRY PI: A 21ST CENTURY
LUNCH BELL
NACS 2018, Branson, MO – Thomas Gölles

2. THOMAS GÖLLES
Head of Modern Workplace Solutions
Co-Organizer of SharePoint Saturday Vienna (#SPSVIE),
Office 365 Meetup Graz, Office 365 Meetup Vienna
http://www.modernworkplacesolutions.rocks/
@thomyg

3. MAKE SURE TO THANK OUR SPONSORS!
WEAR YOUR NACS WRISTBAND FOR
LUNCH SPECIALS ON BRANSON LANDING
PLAY VENDOR BINGO FOR YOUR
CHANCE TO WIN A i7 16GB 1TB SURFACE
BOOK

4. TODAY‘S AGENDA
1 Little bit of background 2 “Bussiness Problem”
3 Tech geek out 4 DEMO

5. OUR OFFICE IN GRAZ, AUSTRIA

6. THE VIEW I USED TO HAVE …

7. EVERYDAY AT AROUND 10:45 …

8. SANDWICH TIME

9. BUT I GOT A NEW OFFICE AND A NEW VIEW…

10. POSSIBLE SOLUTIONS
Create a recurrent event in Outlook
“Ping” a colleague everyday
Stand up and have a look
Use technology to solve the problem

11. MOCADESYMO
Penny: Well, you know, it’s the Cheesecake Factory. People order
cheesecake, and I bring it to them.
Leonard: So, you sort of act as a Carbohydrate Delivery System.
Penny: Yeah, call it whatever you want, I get minimum wage.
Mobile Monitor

12. FRONT DOOR VIEW

13. MOCADESYMO

14. ARCHITECTURE

15. MAKE SURE TO
ATTEND THE
ATTENDEE PARTY
FRIDAY NIGHT AT
7:30PM AT BLACK
OAK GRILL FOR
GREATE LIVE
MUSIC
GET YOUR FREE
DRINK TICKETS AT
THE END OF THE
DAY SESSION ON
FRIDAY

16. CHATBOTS

17. Chatbots 101
“a computer program designed to have a
conversation with a human being, especially over the
internet” *
* https://dictionary.cambridge.org/dictionary/english/chatbot

18. “HOW DO YOU PERCEIVE COMPANIES THAT OFFER A CHATBOT?”
https://insights.ubisend.com/2017-chatbot-report

19. Microsoft Bot Framework

20. COGNITIVE SERVICES

23. Yes
Similar
image
Query
image

24.  Researchers took a traditional machine learning approach
• Example: HoG Detectors
- Histogram of oriented
gradients (HoG) features
- Sliding window detector
- SVM Classifier
- Very fast OpenCV
implementation (<100ms)

25. Deep Neural Network for Computer Vision
cat? YES
dog? NO
car? NO
Convolutional Layers
Fully
Connected
Layers
Complex Objects
& Scenes
(people, animals,
cars, beach
scene, etc.)
Image
Low-Level Features
(lines, edges,
color fields, etc.)
High-Level Features
(corners, contours,
simple shapes)
Object Parts
(wheels, faces,
windows, etc.)

26. TECH GEEK OUT

27. AZURE / RASPBERRY PI

28. RASPBERRY PI + CAMERA MODULE

29. PARTS

30. PI SETUP & TOOLS
Runs Raspbian Jessie
Lan/Wlan connection
raspistill for taking
images
mogrify for cropping
the image
Azure CLI 2.0
curl

31. AZURE COMPONENTS
Blob
Storage
Azure
Function
Custom
Vision API
Storage
Tables

32. BLOB STORAGE

33. AZURE FUNCTION
• Min. Visual Studio 2017
Update 2
• Usage of custom library
• Including PnP library
• Gets triggered by the
curl request from the Pi
• Takes the image from
the blob storage

34. AZURE FUNCTION
• Calls Custom Vision API
• Gets back the prediction
result from http request
• Stores data to log and
state tables
• Informs users in Teams
by calling a connector if
prediction is above a
certain threshold
• uploads pictures to a
SharePoint Portal

35. CUSTOM VISION API
• Project with sample data
• Start with at least 30
images
• Train the models
• Use live data to learn in
iterations
• Be prepared to
understand prediction
results
• Getting above 95% with
~450 images now

36. NORMAL STATE

37. 98,53% TRUE

38. 100% TRUE

39. 99,9% TRUE

40. Log table with every call
and result from the
Custom Vision API
State Table with
timestamps at arriving and
leaving for easier
integration of other
systems in the future
(Cortana, Alexa, bot
framework)
STORAGE TABLES

41. TEAMS CONNECTOR

42. LOG OUTPUT
• Basic incoming
webhook connector in
Teams
• Easy to test with
Powershell request :
Invoke-WebRequest -Uri
http://localhost:7071/api
/FoodTruckMonitorFunc
?image=https://snackcar
monitor.blob.core.windo
ws.net/snackcarimages/
2017-07-26-10-49-
01.png

43. SHAREPOINT PORTAL

44. PICTURE LIBRARY AS A KIND OF ALBUM

45. DEMO TIME

47. DEMO CASE
Blob
Storage
1 2
3
4
5

49. PRIZE DRAWINGS HAPPEN AFTER THE END OF THE DAY SESSIONS ON FRIDAY AND
SATURDAY. YOU MUST ATTEND THE END OF THE DAY SESSION TO WIN.