In this session, you will learn how to build an Internet of Things application that can monitor and control devices via voice and understand step by step building applications based on the following parts: Dialogflow platform, Firebase Realtime Database, Actions on Google platform and connect microcontroller (Arduino, ESP8266, ESP32,...) to Firebase Realtime Database.

1. Internet of Things and Google
Assistant
Le Cong Vinh Khai
TAPIT Co., Ltd Director
IoT R&D Leader
1
SMARTHOME ACTIONS

2. CONTENT
 About us
 Internet of Things and VUIs
 Why Google Assistant?
 Actions on Google: Smart home Actions
 Steps
2

3. 3
About us
Name: Lê Công Vĩnh Khải
Date of Birth: December 4, 1995
Work: IoTs R&D Leader at TAPIT Co., Ltd
Email: vinhkhai95@gmail.com
Phone: +84787412885
Experience: More than 2 years experience in IoTs
research.

4. 4
CÔNG TY TNHH KỸ THUẬT TAPIT
MST: 0401918051
SĐT: 0981001119
Website: https://tapit.vn
Địa chỉ: 146 Trương Định, TP.Đà Nẵng
About us
Internet of Things
Mr. Lê Công Vĩnh Khải
Mr. Ngô Văn Trung
Embedded System
Ms. Trần Thụy Ngọc Hằng
Mr. Đậu Đức Thắng
Mr. Huỳnh Ngọc Thương
Hardware Design
Mr. Ngô Thanh Liêm
Mr. Nguyễn Huỳnh Nhật Thương
CỘNG ĐỒNG KỸ THUẬT TAPIT
300+ thành viên
+ 6 trường đại học từ TP. Đà Nẵng, TP. Hà Nội, TP. Hồ Chí Minh
+ 24% kỹ sư, 19% sinh viên năm 5, 27% sinh viên năm 4, 23% sinh viên năm 3,...
+ 10 chuyên ngành, lĩnh vực khác nhau
Training
-Vi điều khiển: STM32,
MSP430, Arduino.
- Internet of Things.
- C/C++
fb.com/tapit.vn0981001119 https://tapit.vntapitlrs@gmail.com

5. Internet of Things and VUIs
5
VUIs: Voice User Interfaces The Next Disruption

6. Alexa
Google Assistant
Siri
Cortana Bixby
6

7. 7
Why is Google Assistant?
Asia Pacific

8. 8

9. 9
Compatibility

10. Sonoff Philips Hue
Belkin WeMo Lutron
10
Broadlink
Xiaomi

11. Actions
on
Google
Smart Home Actions
Conversation Actions
11
Actions on Google
 Smart Home Actions: Control IoT devices (connect, query and
control through existing cloud infrastructure).
 Conversation Actions: Invoke Actions to build two-way interactive
conversations between users and assistant (News, Shopping, Quiz
Game, IoTs…)

12. Smart home
Actions
Sonoff
Philips Hue
Belkin WeMo
Lutron
12
Broadlink
Xiaomi

13. 13
Smart homeActions
Google Home App Manufacturer/Personal Cloud
auth/token server
smarthome server
2. SYNC intent
3.EXEC/QUERY intent
Request
Response
Response
Request
1. Account linking
OAuth Credentials
 SYNC: Requests the list of devices that the user has connected and are
available for use.
 EXEC: Requests a command to execute on smart home devices
“Ok Google, change the TAPIT light to blue !”
 QUERY: Queries for the current states of devices
“Hey Google, is the TAPIT fan on?”

14. Flow for device setup
14

15. Google Assistant
15
Flow for execution/query
Device Cloud Service
HomeGraph
EXEC intent
QUERY intent
Response
User say

16. 16
Device definition
DEVICE DEVICE TYPE DEVICE TRAITS
Light action.devices.types.LIGHT action.devices.traits.Brightness
action.devices.traits.ColorSetting
action.devices.traits.OnOff
Fan action.devices.types.FAN action.devices.traits.OnOff
action.devices.traits.FanSpeed
action.devices.traits.Modes
action.devices.traits.Toggles
Washer action.devices.types.WASHER action.devices.traits.OnOff
action.devices.traits.StartStop
action.devices.traits.Modes
action.devices.traits.Toggles
action.devices.traits.RunCycle
………. …………… ……………..
Supported more than 20 different device types

17. 17
SYNC intent for LIGHT
{
"requestId": "ff36a3cc-…",
"inputs": [{
"intent":"action.devices.SYNC"
}]
}
{
"requestId": "ff36a3cc….",
"payload": {
"agentUserId": "1836",
"devices": [
{
"id": "123",
"type": "action.devices.types.LIGHT",
traits: [
“action.devices.traits.OnOff”,
“action.devices.traits.Brightness”,
“action.devices.traits.ColorSetting”
],
“name”: {
name: ‘tapit light',
},
……………….
}
Request Response

18. 18
EXEC intent for LIGHT
{
"requestId": "ff36a3cc-….",
"inputs": [{
"intent": "action.devices.EXECUTE",
"payload": {
"commands": [{
"devices": [{
"id": "123"
}],
"execution": [{
"command": "action.devices.commands.OnOff",
"params": {
"on": true
}
}]
}
{
"requestId": "ff36a3cc-…",
"payload": {
"commands": [
{
"ids": [
"123"
],
"status": "SUCCESS",
"states": {
"on": true,
"online": true
}
}
]
}
}
Request Response

19. 19
QUERY intent for LIGHT
{
"requestId": "ff36a3cc…",
"inputs": [{
"intent": “action.devices.QUERY”,
"payload": {
"devices": [{
"id": "123",
}]
}
}]
}
{
"requestId": "ff36a3cc-…",
"payload": {
"devices": {
"123": {
"on": true,
"online": true
}
}
}
}
Request Response

20. 2020
Steps
1. Set up an server for Account Linking (AUTH, TOKEN) and
Smart home intents (SYNC, EXEC, QUERY).
2. Set up database for devices state storage
3. Create an Actions on Google developer project.
4. Provide server URL for requesting.
Using Cloud Functions for Firebase
Using Firebase Realtime Database

21. 21