Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Building Conversational Experiences with Actions on Google

401 vues

Publié le

Since we first started using computers, we have dreamt of interacting with them in a more natural, human-like way using spoken language. Advances in machine learning and artificial intelligence have spiked an interest in natural language processing, and building conversational interfaces no longer is a dream, but a reality every developer can use in their own applications.

The problem of recognising spoken input has been largely solved, and the next challenge is how to build compelling user experiences that are modelled after natural human conversation.

In this talk, I will give an introduction to the Google Assistant and how to use Actions on Google to build conversational interfaces for devices with and without screens. We will take a look at how API.AI makes implementing natural language understanding feasible even for non-linguists, and discuss how you can integrate agents with existing backends. In conversation, it's all about being natural, so we will also look at how to create agents that feel like real characters.

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Building Conversational Experiences with Actions on Google

  1. 1. Extending the Google Assistant with Actions on Google +PeterFriese @peterfriese Developer Advocate, Google
  2. 2. “In the next 10 years, we’ll shift to a world that is AI-first” — Sundar Pichai, CEO Google
  3. 3. Computing becomes universally available
  4. 4. Human computer interaction becomes more intuitive
  5. 5. Products with AI will offer more intelligent, delightful experiences
  6. 6. Mobile-first changed our world in less than 10 years
  7. 7. The Google Assistant — A conversation between you and Google that helps you get things done in your world. Google Home/Mobile device — The surface to interact with the Assistant. Actions on Google — How developers can extend the Assistant (via Assistant apps)
  8. 8. Sorry, but it looks like you missed the last session for Android but you can try to catch it online after Google I/O! Sure! Here’s Google IO 17. Ok Google, let’s talk to Google IO 17 Enter Earcon Exit Earcon Hey there. I have all kinds of info on I/O, from the keynote to the sessions you can catch online. Which of these do you want to know about? When is the next Android talk?
  9. 9. How does it work? Design Develop Discover
  10. 10. Assistant app { conversation api request } { conversation api response } user input app response
  11. 11. Ok Google, let’s talk to Google IO 17 I want to know about Machine Learning Sure, here’s Google IO 17 Hi! Welcome to Google IO! The next ML session is... Speech to Text NLP Knowledge Graph ML Ranking User Profile Handle “action.intent.MAIN” Speech to Text Text to Speech ... ... ...
  12. 12. “ Intent Matching — Match and categorize user utterances to an intent. Entity Extraction — Identify key words and phrases spoken by the user. @
  13. 13. Ok Google, let’s talk to Google IO 17 I want to know about Machine Learning Sure, here’s Google IO 17 Hi! Welcome to Google IO! The next ML session is... Speech to Text NLP Knowledge Graph ML Ranking User Profile Convert “action.intent.MAIN” to WELCOME event Speech to Text Text to Speech ... ... ... ... NLP: Intent Matching Entity Extraction
  14. 14. Intent Triggered via a series of “user says” phrases or platform based events Can collect entity values Matched at every turn of conversation
  15. 15. Input and Output Contexts You can require a context to be available before an Intent is enabled Intent can set context to enable other Intents Retain data across intents
  16. 16. Entities (slot-filling) Values that we are trying to capture from the user phrases Can specify a parameter name and a type of value Values can be optional Values can be a list of fixed values
  17. 17. System Entities ● Time ● Date ● Time Period ● Number ● Cardinal ● Ordinal ● Temperature ● Speed ● Volume ● Weight ● Age ● Currency ● Country ● Location ● Language ● Artist ● Music ● ….
  18. 18. Developer Entities
  19. 19. Specify follow up questions if a user doesn’t specify certain values Read out in random order to make it more natural Prompts
  20. 20. Fulfillment
  21. 21. Node.js Client library https://github.com/actions-on-google/actions-on-google-nodejs const App = require('actions-on-google').DialogflowApp; exports.yourApp = (request, response) => { const app = new App({request, response}); console.log('Request headers: ' + JSON.stringify(request.headers)); console.log('Request body: ' + JSON.stringify(request.body)); // Fulfill action business logic function responseHandler (app) { // Complete your fulfillment logic and send a response app.ask('Hello, World!'); } const actionMap = new Map(); actionMap.set('<Dialogflow_action_name>', responseHandler); app.handleRequest(actionMap); };
  22. 22. From ‘app’ to ‘experience’...
  23. 23. Create a persona.
  24. 24. Preserve and reinforce your persona by engaging the user as a separate entity from the Google Assistant. Own it. Hey! This is ___ Welcome to ___ Ready to play ____ Hi! ___ here. Hello. I’m ___ Greetings, human. Welcome back to ___ Hey again. ___ Let’s play ___ Here’s your ___ Brought to you by ___ Hi there, ___ Let’s get started. Ready for your ___ ___, here to… Live from ___ This is ___ What’s up, ___ and more...
  25. 25. <speak> <!-- Must be at the start of the string --> <say-as interpret-as="characters">SSML</say-as> <break time="3s"/>. <audio src="https://example.com/file.mp3"></audio> <prosody rate="slow" pitch="-2st">Can you hear me?</prosody> <emphasis level="moderate">This is important</emphasis> <sub alias="World Wide Web">WWW</sub>. <p><s>This is one.</s><s>This is two.</s></p> </speak> <!-- Must be at the end of the string --> Reinforce it with SSML Speech Synthesis Markup language “S S M L” [3 second pause] [audio file plays] “Can you hear me?” “This is important” World Wide Web [two sentences]
  26. 26. // This function is used to handle the welcome intent // In Dialogflow, the Default Welcome Intent ('input.welcome' action) // In Actions SDK, the 'actions.intent.MAIN' intent function welcome (app) { if (app.getLastSeen()) { app.ask(`Hey you're back...`); } else { app.ask('Welcome to Number Genie!...'); } } Reinforce it with persistence https://developers.google.com/actions/assistant/best-practices
  27. 27. Design for multiple surfaces.
  28. 28. Support different surface capabilities https://developers.google.com/actions/assistant/surface-capabilities AUDIO_OUTPUT SCREEN_OUTPUT
  29. 29. function simpleResponse (app) { app.ask({ speech: 'Howdy! I can tell you fun facts about ' + 'almost any number, like 42. What do you have in mind?', displayText: 'Howdy! I can tell you fun facts about ' + 'almost any number. What do you have in mind?' }); } Support speech and display text https://developers.google.com/actions/assistant/responses Chat text should be a subset of audio
  30. 30. function suggestionChips (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Howdy! I can tell you fun facts ‘ + ‘about almost any number. What number do you have ‘ + ‘in mind?') .addSuggestions(['0', '42', '100', 'Never mind']) ); } Guide the user (suggestion chips) https://developers.google.com/actions/assistant/responses Chip text should be under 25 characters
  31. 31. function basicCard (app) { app.ask(app.buildRichResponse() .addSimpleResponse('Math and prime numbers it is!') .addBasicCard( app.buildBasicCard(`42 is an even composite number. It ` + `is composed of three distinct prime numbers multiplied together. It ` + `has a total of eight divisors. 42 is an abundant number, because the ` + `sum of its proper divisors 54 is greater than itself. To count from ` + `1 to 42 would take you about twenty-one…`) .setTitle('Math & prime numbers') .addButton('Read more') .setImage('https://example.google.com/42.png', 'Image alternate text') ) ); } Display basic cards https://developers.google.com/actions/assistant/responses
  32. 32. Lists and carousels for selection https://developers.google.com/actions/assistant/responses Used for easy selection <10 items Used for comparison <30 items
  33. 33. Know the user.
  34. 34. Updates
  35. 35. Google Home Mobile Device NAME Registered device user’s full name Registered device user’s full name DEVICE_COARSE_LOCATION Zip code and city N/A DEVICE_PRECISE_LOCATION Coordinates and street address Coordinates let permission = app.SupportedPermissions.DEVICE_COARSE_LOCATION; app.askForPermission('To find bookstores near you', permission); Q: "Recommend me a local bookstore" A: "To find bookstores near you, I'll just need to get your zip code from Google. Is that okay?" Ask for information https://developers.google.com/actions/assistant/helpers#user_information
  36. 36. Link an account (OAuth2) https://developers.google.com/actions/identity/ Seamless account linking with Google Sign-in
  37. 37. Transact with the user https://developers.google.com/actions/identity/ Build orders Use Google provided payment instrument Use your payment processor (Stripe, Braintree, Vantiv, more coming) Update order status
  38. 38. Reach users...
  39. 39. In Dialogue Discovery: Explicit Triggering (of Actions) Ok Google, ask Google I/O 17 about what was announced Trigger Phrase App Name Developer Specified Action Preposition Action Phrase Developer Specified Partner Examples: Let’s speak to Domino’s At Akinator Ask Dr. Doggy if dogs can eat chocolate Ok Google, let’s talk to Google I/O 17 Trigger Phrase App Name Developer Specified
  40. 40. Hey Google, I want to play a game.Hey Google, tell me a joke. In Dialogue Discovery: Implicit Triggering Hey Google, I want to work out. Hey Google, what’s the surf report
  41. 41. Discovery in the Google Assistant Directory
  42. 42. Link to your Assistant App From anywhere Share through social media. Promote through your own site or apps. Encourage press to drive traffic to your Assistant app.
  43. 43. What makes a great app
  44. 44. Where do I start?
  45. 45. Actions Console console.actions.google.com
  46. 46. 0. Put away the code 1. Draft a persona 2. Write some dialog
  48. 48. g.co/dev/ActionsDesign
  49. 49. developers.google.com/actions
  50. 50. Action Templates
  51. 51. Actions on Google Developer Community https://g.co/actionsdev Actions on Google Developers https://developers.google.com/actions +PeterFriese @peterfriese Developer Advocate