Smart Home DesignBased On Voice Recognition

Using Galileo and Raspberry PiWritten in Python script language

Semesterial project

Students : Matan Cohen & Sefi CohenSupervisor : Mony Orbach

Spring 2014

Most smart homes today require a remote instead of being hands-off

We want to automate home life

We want to control appliances from afar without remotes

Smart home products are expensive

Smart home products aren’t extendible

Motivation

Project GoalsTo build an infrastructure of an interactive home

It’ll need to be hardware extendible – Can add more components in the future easily

It’ll need to be software extendible – Can add more software features easily

To build a prototype which turns on a led with the right command.

Voice recognition based commands Since most products out there require a remote (which is a hassle),

ours will be hands-off.

Communicate with the user – Using TTS we can have the system speak back and give the user indications on actions being performed

To experience designing hardware and software product from scratch.

To get our hands dirty with both hardware and programming at the hardware & software environments.

Project Vision Sketch (for example)

LIVINGROOMINTERNET

INTERNET

INTERNET

BEDROOM

WHEREVER

Raspberry Pi

Raspberry Pi

SERVER(RP or PC)

Galileo

Electric

Blinds

MIC

MIC

Galileo

TV

Hardware EnvironmentRaspberry Pi Galileo

• Based on ARM• Linux based• Open Source• On board GPU• On board sound card• Ethernet

• Based on Intel Quark• Arduino based• Open Source• Ethernet

Software EnvironmentRaspberry Pi - Linux environment

Using Python script language.

Galileo – Linux environmentUsing implemented library of Arduino in Python.

Both are controlled via SSH protocol.

Hardware Architecture

Linked through internet

Software ArchitectureRaspberry Pi - Linux environment

Listener program – record the voice from the microphone and send the wav file to voice recognition unit .

Voice recognition unit – based on WIT learning speech recognition web service online.

RabbitMQ SERVER – Linux environment Server SW– receive and send the analyzed messages from

client to client. Command Parser– parse the received string and send a

signal to the right Galileo to run the right add-on script

Galileo – Arduino environment Add-ons scripts written in python using Arduino library . Scripts like lights and blinds are already implemented.

Configuration fileWe need to initialize the HW component with

information that it needs to function properly.When we add a new component (recording like Rpi or

commands invoker like Galileo) we have to write a “system config” file which contain the following : Mic – insert microphone dry details : Format, Rate,

Channels number and Chunk size. Voice Energy Threshold – which the microphone will

start recording from this energy and above. WIT Key – that serial number is needed for connecting

to the WIT web service. Name – The component's name. It takes part in

routing the message to the right component via RabbitMQ.

Commands – The addons that are supported on this component.

External SW ComponentsRabbitMQ – Server Implementation

There is a lot of different implementation of servers – we choose “Routing” which in this implement every target has a queue of his own and he gets only the messages that’s addressed to him

In the code the RabbitMQ is called “AMQP” (Advanced Message Queuing Protocol) which is the name of the protocol which RabbitMQ implements .

External SW Components cont.wit.ai – Voice recognition web service

enables developers to add a language interface to their app or device in minutes.

It’s a learning system and as one we need first to provide a few examples of queries to get the expected responses.

We use their API to send queries and get the response as structured information back (JSON)

Py Galileo Library – Python library for Galileo board Helps us write code in python that control the pins on

the board like Arduino does.

Main Software ComponentsText To Speech Module – Given a string, create

an audio file using a Google TTS service that says that string aloud. Plays that file.

Wit Query Analyzer – using WIT web service. Given an audio recording of a command, the WIT service returns a JSON that can be worked with containing information about what was said.

Client - Manages sending parsed commands to the correct HW component.

Main Software Components cont.

Receiver - Receives a command from another HW components and starts processing it by sending it to the Addon Manager

Addon Manager – when the Receiver gets a command the Addon Manager is responsible for checking the validity of the command and executing the right script

Flow of events - System POVVoice message is received in Raspberry Pi

The message is analyzed by WIT web service online and a json file returns as a string

This json file is wrapped by private object called wit_response – locally on Raspberry Pi

From Rpi the analyzed message is sent to the right target by the RabbitMQ server which send the message according to the target’s name

Flow of events - System POV cont.The target (Galileo or other Rpi) receives the wit

response and transfers it to the add-on manager

The add-on manager that runs on the target checks if (in the system configuration file) the target supports the command. It does this by checking if the "intent" field in the wit response object is found in the "commands" field list in the component's JSON configuration file.

If there is a match the add-on manager executes the right script (and pass to the add-on the original wit response since the wit response includes information needed by the addon itself as well.)

Demonstration Equipment1 Raspberry pi

2 Galileo

1 PC (the SERVER)

1 SD card (for Rpi)

2 mini SD card (for Galileo)

1 USB sound card

3 Ethernet cables

2 USB microphone

1 Speakers

1 Led circuit

1 Buzzer circuit

Demonstration Environment

RP Galileo

LED

MIC

SPK

INTERNETRabbitMQ SERVER

Galileo

BUZZ

SSH Connection over TCP/IP

Ethernet Connection

USB Connected Electric

CircuitElectric Circuit

Starting DemonstrationBurning the Linux version on SD cards for all

the hardware components.Connecting the all the Demo environment as

the sketch in the last slide.Connect the Rpi and Galileo through direct

Ethernet connection.Run the command Nmap from any PC in order

to find the hardware IP’sConnected to all the hardware components

through SSH protocol over TCP/IP.

Starting Demonstration cont.Write a configuration file for each

component : Measure the energy threshold using script. Insert the Microphone factory properties . Inset the WIT key. Give a name to the component. Enter the commands that are available for this

“room”/HW component.

Execute the RabbitMQ server on PC or on one of the Raspberry Pi and “tell” the other components his IP.

Execute on each Raspberry Pi in the all system the client “audio_client” program.

Execute on each Galileo the “receiver_client” program.

FAQWhy python ?

It’s a strong language with all the libraries we need. It’s easier to write code in.

Why do we need the server in the middle ? Because we want to make Hardware extendible infrastructure and we

need a server to control all the units in the house .

How can we configure new microphone ? In the system configuration file we can edit the mic factory settings

(format, rate, channels number and chunk size).

How can we configure energy threshold in the config file ? Special script written for that matter ,named energy_threshold.py,

which measure the silence and loud energies in the room (will be shown in demo).

FAQ cont.What about the delay of the all system?

We add to our software “performance counters” for getting information about the time of each action.

The voice recognition proccess (using WIT) is taken between 6 to 12 seconds depend on the network speed and the load on WIT server.

The sketch upload to the right galileo device is taken between 1 to 2 seconds.

How can we do the initialize step hands off ? At our home we could give the server static IP and write the IP in

the configuration file so that all the other devices will know who is the server.

Then we need to define in each device the right scripts that’s needs to be uploaded from the start.

Future UpgradesMake the voice recognition SW run locally

It will shorten the delay significantly. “Sphinx” – library that implement voice recognition locally.

Use Zero-configuration networking In our project its important that each device is getting an IP

address. In this methodology DHCP server is not necessary for

network configuration and there is no need in manual operator intervention or other special configuration servers.

Make the all system wireless

Future Possible ProjectsAdd camera and use facial recognition

algorithms If its not someone familiar turn on the alarm. If its someone familiar turn on the A/C and his favorite

music.

Add a web application of user interface Easy to configure hardware components. Easy to add software add-ons. Control away from home will be enabled.

Questions?

Thank you all very much!!!