Extend the Google Assistant with Actions On Google.

1. Extending the
Google Assistant

2. 2
• How Am I?
• Google Assistant
• What is it?
• How Does it work?
• Build a Natural Language Understanding(NLU) Model
• What is a NLU model?
• Overview of DialogFlow.ai (former Api.ai)
• Fulfillment with Actions-On-Google
• Backend with Cloud Functions for Firebase
• Development Environment with Ngrok
• Ask the user
• Know more about the user
• Reach your users
• Hands-on
• Let's Build a rental Shisha app!
Agenda

3. How Am I ?
• Ndjido Ardo BAR : Lead Data Scientist @ Davidson Consulting UAE
• Background : Research In Mathematics
• Now: Working in @RTA as a Consultant
• Past:
• Working in @DU Telecom as a Consultant
• Worked @AXA (Paris)
• Worked @BearingPoint Hypercube
• Co-Founder of a StartUp (MLouma)
• Worked @Pasteur Institute: Involved in BioStatistical Research
@ndjido
3

4. The Google Assistant
4

5. The Google Assistant
The new way to talk to Google!
What is it?

6. The Google Assistant
How Does it Work? (1/2)
Source: Google
Actions-On-Google

7. The Google Assistant
How Does it Work? (2/2)
Source: Google

8. Building a Natural
Language Understanding
Model
4

9. Building a Natural Language Understanding Model
What is a NLU model?
TextVoice
I want to rent a mint Shisha for 6pm
NLU model
Intent: Rent Shisha
Entities:
- flavor: Mint
- Time: 6pm

10. Building a Natural Language Understanding Model
DialogFlow Overview
Intent Classification - Match & Categorize user utterances to an intent
Entity Extraction - Identify parameters for the backend logic.

11. Building a Natural Language Understanding Model
DialogFlow Overview: Intent Creation
Intent Name
User saysSystem Entities
Custom Entities
Slots or parameterRequired Parameter

12. Building a Natural Language Understanding Model
DialogFlow Overview: Custom Entity Creation
Entity Name
Entity values

13. Building a Natural Language Understanding Model
DialogFlow Overview: Action, Contexts and Events
Output Context
Input Context
Lifespan
Action Name
Built-in & Custom Event for triggering intent

14. Building a Natural Language Understanding Model
DialogFlow Overview: Fulfillment
Webhook URL with SSL termination

15. Building a Natural Language Understanding Model
DialogFlow Overview: Activate Fulfillment for an Intent
Activated Webhook
Static Text Response

16. Building a Natural Language Understanding Model
DialogFlow Overview: NLU Response in JSON format
Tests

17. Fulfillment with
Actions-On-Google
4

18. Fulfillment with Actions-On-Google
Actions-On-Google Overview
Source: Google

19. Fulfillment with Actions-On-Google
Deployment with Cloud Functions for Firebase
Going Serveless with Cloud Functions
Pub/Sub
Firebase
Cloud Storage
Source: Google
We use Cloud Function Local Simulators in Dev Environment

20. Fulfillment with Actions-On-Google
Development Environment with Ngrok
Source: ngrok.com
Ngrok creates a secured tunnel to your localhost.
Firewall + SSL Termination + tunnel to Internet

21. Fulfillment with Actions-On-Google
Ask the user: On different Surface Capabilities

22. Fulfillment with Actions-On-Google
Ask the user: Speech Synthesis Markup language
Make the conversation sound more natural with SSML
https://developers.google.com/actions/reference/ssml

23. Fulfillment with Actions-On-Google
Ask the user: Summarize audio with text
Use tell() to send a message and terminate the conversation

24. Fulfillment with Actions-On-Google
Ask the user: Use Suggestion chips

25. Fulfillment with Actions-On-Google
Ask the user: Guide the user with lists
<=30 items

26. Fulfillment with Actions-On-Google
Ask the user: Use Cards to show more information

27. Fulfillment with Actions-On-Google
Ask the user: Use Carousels to show more information
<=10 items

28. Fulfillment with Actions-On-Google
Ask the user: Ask for more information with PERMISSIONS
https://developers.google.com/actions/reference/rest/Shared.Types/Permission

29. Fulfillment with Actions-On-Google
Account Linking with OAUTH
Automatically Sign Up Users with Streamlined Identity Flows
https://developers.google.com/actions/identity/oauth2-assertion-flow

30. Fulfillment with Actions-On-Google
Transactions
Source: Google
Leverage millions of Payment Card available in Google

31. Reach your Users
with
Dialog Discovery
4

32. Reach your Users
Explicit Triggering => Include App Name
Single Point Of Entry

33. Reach your Users
Implicit Triggering => Trigger with "user sayings"
Multiple Points Of Entry
"Users' sayings" in the added intents can trigger your assistant

34. Hands-On
4

35. Thank You!
Questions ?
@ndjido