Voice Assistants and IoT AutomationIntel, Software and Services GroupDaniel Holmlund - @agnathan

Voice AssistantsAnd

IoT Automation

Agenda

➔ Overview of Cloud and Edge Components needed to enable Voice IoT Automation

➔ Installing Voice Assistants on the Intel® IoT Gateway◆ Amazon Alexa*, Google Assistant* and Mycroft.ai*

➔ Rendering Voice Data into Intent ◆ Alexa Voice Service*, Google Actions*, Mycroft Intent*

➔ Intent Fulfillment◆ Amazon Lambda*, Google Cloud Functions*◆ Building Our Own Intent Fulfilment Service on the Intel* IoT Gateway

➔ Build Our Own IoT Automation Service on the Intel* IoT Gateway◆ Discussion about State, Device Shadows, Flux Pattern

Voice and Automation Overview Diagram

Google* Actions

Alexa* Voice Services

Google* Cloud Functions

Amazon* Lambda

Voice Fulfillment Service

Edge Network

Cloud

Automation Service

Intel IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant*

Amazon Alexa*

Mycroft.ai*

A voice to IoT device automation workflow can be built in the cloud or it can be built entirely on the edge network.

Questions to Consider

➔ Do I want to enable every IoT Device with a Voice Assistant or just a Gateway?

➔ How do I represent the state of an IoT device? How do I change the state of an IoT device with voice?

➔ How do I build a voice to IoT automation workflow in the cloud?

➔ How do I build a voice to IoT automation workflow on the local edge network?

➔ What are the tradeoffs building in the cloud vs on the edge?

➔ How can I build an IoT Automation service?

Installing Voice Assistants on the Intel® IoT Gateway

Demo: Voice Assistants Installed on Intel® IoT Gateway

Alexa Voice Service* Installation

➔ Alexa Voice Service* allows you to install the Alexa voice assistant on a device.

➔ Alexa Skill Kit* enables you to write a program that enables Alexa Voice Service to respond to new voice commands or questions.

Register a New Device

To Create a New Device on the Alexa Voice Service* enter the following:

➔ Device Information - Create an identity for your Alexa* enabled device.

➔ Security Profile - how Amazon* identifies your device. This binds your security profile to an oauth2 id and secret

◆ Security Profile ID

◆ Client ID

◆ Client Secret

➔ Device Details - Image, Category, Is it going to be a publically sold device, etc ...

Amazon Alexa* Installation on the Intel® IoT Gateway

Amazon Provides Access to the Alexa Voice Assistant* through Amazon Voice Services*

● Register for an Amazon Developer Account○ Create an Alexa Voice Service Project○ Create a device and security profile

■ ProductID (Device Type ID),

■ ClientID

■ ClientSecret

● Setup on the Intel IoT Gateway○ git clone https://github.com/alexa/alexa-avs-sample-app.git○ Enter the ProductID, ClientID and ClientSecret in the install script○ Install VLC for audio API dependencies○ Run generate.sh to create self-signed certificates○ Run the sample Java application that communicates to AVS○ Run the NodeJS Companion Application which listens for a response

from AVShttps://developer.amazon.com/alexa-voice-service

Intel® IoT Gateway

* Voice Agents are installed on the Gateway

Amazon Alexa*

Google Assistant* Installation on the Intel® IoT Gateway

● Setup on the Google Cloud Platform○ Create a Google Developer Project○ Enable the Google Assistant API on the project○ Enable the Billing API on the project (set a budget)○ Create an OAuth Client ID and download the client_secret file in

JSON format.

● Local Setup Option 1: Google Assistant library for Python$ sudo apt-get install python3-dev python3-venv

$ python3 -m venv env

$ env/bin/python -m pip install --upgrade pip setuptools

$ source env/bin/activate

$ (env) python -m pip install --upgrade google-assistant-library

$ (env) python -m pip install --upgrade google-auth-oauthlib[tool]

$ google-oauthlib-tool --client-secrets /home/pi/client_secret_client-id.json

--scope https://www.googleapis.com/auth/assistant-sdk-prototype --save

--headless

$ google-assistant-demo

● Local Setup Option 2 - Google Assistant gRPC API - exposes a low

level API which directly manipulates the audio bytes of an Assistant request

and response

https://developers.google.com/assistant/sdk/prototype/getting-started-pi-python/config-dev-project-and-account

Intel® IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant*

Resolving Voice to Intent

Services that Render Voice to Intent

Google Actions

AmazonAlexa

Service

Edge Network

Cloud

Intel® IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant*

Amazon Alexa*

Mycroft.ai*

➔ Mycroft.ai does not require a cloud component. Voice Fulfillment can happen entirely on a local edge network.

➔ This does not mean that a developer can forego encryption and good security practices.

➔ Both Amazon and Google require a cloud and local software to enable their voice assistants.

➔ Secure communications must be setup correctly between the cloud and the gateway.

➔ Local software is activated with a keyword, voice is recorded and uploaded to a cloud service that converts the voice to an API call.

Communication between Intel® IoT Gateway and Cloud Voice Services

Google Actions

AmazonAlexa

Service

Edge Network

Cloud

Intel® IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant*

Amazon Alexa*

Mycroft.ai*

➔ Amazon uses HTTPSv2 to communicate audio data

➔ Google uses gRPC, open source high performance RPC framework

➔ Both Amazon and Google use Oauth2 client ID and secrets to authenticate and authorize a device to their respective cloud platforms

➔ You must implement this oauth2 interactive or use their client libraries to do so.

➔ An Audio file with the voice request is sent to the cloud.

➔ Google Actions and Alexa Voice Services use Artificial Intelligence techniques to resolve the audio into an Intent.

➔ Once resolved to an Intent, the Intent is forwarded to a Intent Fulfillment Service

HTTPSv2

gRPC

Training a Cloud Service to Understand Voice

➔ Voice data is transcribed to textual data.

➔ The developer provides a set of example phrases that match each intent

➔ The Cloud Service trains a set of classifiers to match example phrases to intents

➔ Each Cloud Provider uses their own combination of deep learning neural networks, voice models and classifiers

Example Phrases Matching Intent

Please turn on the lights. smarthome.lights.brightness.on

Are the lights on in the family room? smarthome.lights.brightness.check

Please turn the brightness of the lights to 50%.

smarthome.lights.brightness.set

Please turn the lights to soft white. smarthome.lights.color.set

Please turn the lights off. smarthome.lights.brightness.off

Set the lights to pink! smarthome.lights.color.set

What is an Intent?

● Has a set of examples phrases that identify the intent

● May require a context to be set before text can be match to it.

● May trigger an event.

● May be require

Training a Cloud Service to Understand Voice

➔ Example Phrases are matched to intents

➔ Each intent has a series of “Entities” or “Slots” which are keywords or phrases that are converted to parameters

➔ Voice Assistant may ask follow up questions to get all the intent’s required parameters

➔ Parameters are labeled in a JSON object and sent to the Fulfillment Service over HTTPS

Example Phrases Matching “Entities” / “Slots”

Please turn on the lights. Green words are $Device

Are the lights on in the family room? Blue indicates $Color

Please turn the brightness of the lights to 50%%.

Pink matches $Room

Please turn the lights to soft white. Red shows a $State

Mycroft Adapt Intent Parser*

The Adapt Intent Parser is an open source software

library for converting natural language into machine

readable data structures. Adapt is lightweight and

streamlined and is designed to run on devices with

limited computing resources. Adapt takes in natural

language and outputs a data structure that includes

the intent, a match probability, a tagged list of

entities. The software was developed at Mycroft AI by

a team led by Sean Fitzgerald, formerly one of the

developers of both Siri and Amazon Echo.

➔ https://adapt.mycroft.ai/start/examples

➔ https://github.com/MycroftAI

Google* owns Api.ai* - a tool for building voice interactions

Alexa Skill* Development Demo

Intent Fulfillment and Serverless Functions

Intent Fulfillment: Voice and Automation Overview Diagram

Google* Actions

Alexa* Voice Services

Google* Cloud Functions

Amazon* Lambda

Voice Fulfillment Service

Edge Network

Cloud

Automation Service

Intel® IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant*

Amazon Alexa*

Mycroft.ai*

➔ The Fulfillment stage can be continued in the cloud or send back to the Intel IoT Gateway on the local edge network.

➔ In order for Google Actions or Alexa Voice Services to contact your Voice Fulfillment Service it must be accessible on the Internet or over a VPN.

Serverless

Serverless Documentation

Alexa Lambda*

Google Cloud Functions* Hello World Voice Fulfillment Template

MicroserviceFulfillment

From Google Actions*, Alexa Voice Service* or Mycroft.ai* an HTTPS request is received by the Intel® IoT Gateway

The fulfillment function will create an JSON Object that represents the state of the light bulb.

The JSON looks like:

{sensor_id: ‘lightbulb’, value: ’on’, timestamp: Date.now()}

The JSON is published over MQTT-TLS

Need screenshot of my code

Let’s Build Our Own Automation Service

Voice and Automation Overview Diagram

Google* Actions

Alexa* Voice Services

Google* Cloud Functions

Amazon* Lambda

Voice Fulfillment Service

Edge Network

Cloud

Automation Service

Intel IoT Gateway

* Voice Agents are installed on the Gateway

Google Assistant

Amazon Alexa

Mycroft.ai

➔ The Fulfillment stage can be continued in the cloud or send back to the Intel IoT Gateway on the local edge network.

➔ In order for Google Actions or Alexa Voice Services to contact your Voice Fulfillment Service it must be accessible on the Internet or over a VPN.

29

AutomationIP Network

Upstream connection3G/4G/5G or Internet

Secure the hardware, operating system, data and network.

AutomationDataProcessing

Intel IoT GatewaySensor data and actuator commands are communicated over MQTT-TLS

Mosquitto broker running MQTT-TLS on the edge network on the Intel® IoT Gateway.

Automation Rules are stored in a MongoDB database on the Intel® IoT Gateway

Local Fulfillment Functions emits a JSON object that contains the state of the IoT Device that the automation.

ManagementSecurity

Administrative Interface

What is a Trigger?

A trigger is a predicate (condition) function and an action function that implements the rule based business logic on the edge network.

The predicate function returns a true or false, a “Go” or “No Go”

In our implementation, the stash always holds the last sensor reading from each sensor.

Device Triggers Actions

There are many possible events that an action function could trigger.● Write Data to the LCD screen● Restart the device● Execute an application● Run a script● Request data item from the device● Send an SMS messages to an administrator.● Send time synchronization ● Send an alert to the cloud

Creating a Connection to the MQTT Broker

A connection URI looks like mqtt://192.168.1.1/ NodeJS has a NPM package for MQTT.

var mqtt = require(‘mqtt’) will import the package

MQTT has an event driven loop that allows you to customize your response to different events.

On the Connect Event, we log that we connected and subscribe to the “announcement” topic

the “+” matches any string. We are subscribing to all sensors that publish data be subscribing to “sensors/+/data”

33

The Context Object

This object holds variables and libraries that will be made accessible to the condition and trigger functions.

If you want to use HTTP or MQTT in a trigger define them in the context object.

Rules Engines

For each incoming piece of data

filter the list of rules by the sensor that send the incoming data

and evaluate the condition function with the context

if it returns true then execute the trigger function

Condition Functions

Conditions Functions, also called predicate functions, always return TRUE or FALSE.

Example MQTT Trigger Function

Example Trigger Function

Publish an JSON error report to the MQTT topic sensors/temperature/errors

mqttClient and ErrorModel are part of the Context Object and now are accessible under the this object

Create a new MongoDB document called error.

Save error to the database

Be sure to put the parentheses around the entire function

Example HTTP Trigger Function

Example Trigger Function

Create an HTTP POST request for the Intel Edison with the LCD.

Send the HTTP request with the this.http object.

Write two functions to handle the success and error conditions.

Be sure to put the parentheses around the entire function

References

● https://www.xda-developers.com/how-to-get-google-assistant-on-your-windows-mac-or-linux-machine/

● https://inteliot.withthebest.com/#/insights/59275dec9e27000004ac3eca

● https://en.wikipedia.org/wiki/Long_short-term_memory

● https://github.com/alexa/alexa-avs-sample-app

● https://github.com/alexa/alexa-avs-sample-app/wiki/Raspberry-Pi

● https://github.com/alexa/alexa-avs-sample-app/wiki/Create-Security-Profile

More about Voice Recognizers

https://developer.amazon.com/public/solutions/alexa/alexa-voice-service/reference/speechrecognizer#overview