October 4, 2016

Santa Clara Convention Center

Mission City Ballroom

Solutions for playing high-quality

music and voice over BLE

Roberto Sannino

Audio Over Bluetooth Smart: Scenarios 2

Voice controlled TV remoteAudio Entertainment & Gaming

Smart Home Wearables

Voice Terminal

• Audio capture & render

• Signal processing

• Low power

• Constained geometry

Voice & data

GatewaySeamless connectivity

BlueVoice: IoT Audio Augmented 3

Cloud

• Natural Language Processing

• Dialogue Management

• ServicesPlay Music

Control Lighting, heating, …

News, sport, traffic, weather, …

Answer questions, create to-do lists, shopping lists, …

Place orders online, use other online services: taxi, pizza, …

A BlueVoice to Cloud Architecture

Integrated Terminal

Audio Front End

Signal Processing

Communication

& 3rd Party SW libs

Cloud-based

ServicesGateway

4

BlueVoice at a Glance 5

MEMS Microphone Microcontroller BlueNRG RF connectivity

The ST HW and SW solution for

Ultra low-power Voice streaming over Bluetooth Low-Energy

STM32 Signal Processing and Application Firmware & Software

BLUEVOICELINK BLE and microphone reference application based on STM32Cube

OSXBLUEVOICE Voice-over-BLE vendor-specific profile library for STM32 and BlueNRG

BLUEMICROSYSTEM Voice and Sensor data over a BLE link to an Android - iOS Smartphone App

BlueMS, BlueST-SDK Bluetooth Smart and Sensors Technology Application for Android and iOS

BlueVoice Mapping over StandardBluetooth 4.0 protocol stack

6

BlueVoice Vendor Specific Profile

Audio processing

Generic Access Profile (GAP)

Generic Attribute Profile (GATT)

Attribute Protocol Security Manager

Logical Link Control and Adaptation Protocol

Host-Controller Interface

Link Layer Direct Test Mode

Physical Layer

Bluetooth Low Energy Stack

GAP configuration

GATT configuration

Application: BlueVoice Profile

(vendor specific)

Central Unit

(Master)

Peripheral

Unit

(Slave)

Client Server

ClientServer

Service

Characteristic

Characteristic

Descriptor

Audio is

exported as

a Service

Service

Characteristic

Characteristic

Descriptor

BlueVoice Server Attributes

Characterstic Value

Compressed audio data

Audio Characteristic

Service Declaration

• BVS UUID

• Type: primary

Characteristic Declaration

• AudioData UUID

• Char properties:

• Notification

• Maximum value length: 20 B

Characteristic Declaraton

Characteristic Value

Characterstic Value

Synchronization data

Sync Characteristic

Characteristic Declaration

• SyncData UUID

• Char properties:

• Notification

• Maximum value length: 6 B

Characteristic Declaraton

Characteristic Value

Reference chart

BlueVoice Service

Service

Characteristic

Characteristic

Descriptor

Server

7

BlueVoice Packet Exchange Sequence

Advertise

Connect Request

Notification

Notification

Notification

Advertise

Advertisement

ModeDiscovery

Mode

Central Unit(Master)

Peripheral

Unit(Slave)

Connection created

Notification

Notification

Notification

Peripheral-to-

Central

communication

Central-to-Peripheral

communication

......

No response is required for

Notifications:

• OK for Audio streaming

• Optimal Bandwidth

exploitation

8

BlueVoice Architecture: Audio processing and transmission

PDM to PCM

conversion

PCMPDMAudio

Compression

Raw Data

Server - TX

Audio

Acquisition

symmetrical architecture for bi-directional communication

Audio

Decompression

Serial Audio Out

USB, I2S, …

Client - RX

9

BlueVoice Development Kit

BlueVoice on open.Audio 10

Voice Gateway

Cloud

BlueVoice Terminal

• Microphone Acquisition

• PDM to PCM conversion: 8 and 16 KHz

• ADPCM compression: 4 bit per sample

• BTLE streaming: 32 and 64 kbps

SensorTile

BlueCoin

BlueMS app

Software IP and ST Eco-systemOpen Software Design Environment

11

Algorithms and system demonstrators for the Internet of Things.

Unleashing the power of embedded software

Bring your ideas to now!

BlueMicroSystem

STM32 ODESTM32 Nucleo

development

boards

STM32 Nucleo

expansion

boards

STM32 Cube

software

STM32 Cube

expansion

software

Software libraries

BlueVoiceLink

SmartAcoustics

BlueMSAndroid & iOS

Example Projects

Open.AUDIO OSXBlueVoiceVoice over BLE Vendor Specific Profile library

12

PDM to PCM

conversion

PCMPDMADPCM Audio

Compression

ADPCM Data

Server - TX

Microphone

Acquisition

ADPCM Audio

Decompression

Serial Audio Out

USB, I2S, …

Client - RX

16k or 8k sample/s

4 bit/sample

16bit/sample

@16kHz or

@ 8kHz

OSXBlueVoice Speech Tx library

Open Software

Expansion

Open.AUDIO

OSXBlueVoiceThe BlueVoice Library

OSXBlueVoice Speech Rx library

ADPCM Data

16k or 8k sample/s

4 bit/sample

Audio BLE Streaming 13

Connection

Event

10ms t

Audio Data Packet, 20 bytes

160ms

Connection interval

Connection

Event

…

ADPCM Synch interval

Sync Data Packet, 6 bytes

… ………

Data streaming is implemented as periodical data exchange

• Connection Event: data packet exchange sequence

• Connection Interval: the time between two consecutive Connection Events

Radio

Idle

Connection

Event

Connection

Event

Connection

Event

Low-power state between connection events

Voice Streaming

• Audio Format: PCM 16 bit @ 16 kHz

• ADPCM Compression @ 64 Kbps: Low

latency and low complexity

• Side Information at low data rate

ehnances error resilience

OSXBLUEVOICE v2.0.0 Library 14

OSXBLUEVOICE 2.0.0

Working mode:• TRANSMITTER

• RECEIVER

• HALF-DUPLEX

Configurable:

• Audio Sampling Frequency: 8 or 16 kHz

• Audio input channel select

Audio input buffer: 1, 2, 5 or 10 ms of PCM

audio data.

BLE service and

characteristic

modularity

Compatible with Android and iOS app: STBlueMS

osxBlueVoice 2.0.0 Library 15

osxBlueVoice 2.0.0

Features

Compiled for different

ARM Cortex core:• M0+

• M4

STM32 Example code for:• STM32F4

• STM32L4

• STM32L0

ST development Platforms:• STM32 Nucleo

• SensorTile

• BlueCoin

Hardware CompatibilitySTM32 application project

osxBlueVoice 2.0.0 Library Architecture 16

Input

ch1 ch2 ch1 ch2 ch1 ch2 ch1 ch2……

PCM buffer:- 1, 2, 5 or 10 ms audio

- One or more audio channel

- Audio sampling frequency: 8 or 16 kHz.

- Total audio channels acquired.

- Audio Channel chosen: one of the

acquired channels (1, 2 or … n).

- Service and characteristics configuration

ConfigurationosxBlueVoice Library 2.0.0

Service and characteristics creation/configuration

BLE packetization and

sending

BLE packet parsing and

ADPCM decompression

Audio parameters configuration

BLE packets:

- Audio data

20 Bytes

- Sync data

6 Bytes

Output

Audio send application

TX Control

Audio receive application

RX Control

SendData ready

GATT

Notification

BLE Event

ADPCM

compression

Internal double

buffering

Data Control

Input

Output

BlueVoiceLink reference application17

www.st.com/bluevoicelink

BLUEVOICELINK1

osxBlueVoice 2.0.0

BLUEVOICELINK1Half-Duplex communication based on STM32 Open development Environment

Audio configuration

8 kHz,16 kHz4:1 ADPCM

USB USB

BlueVoiceLink1 2.0.0 - Peripheral

osxBlueVoice 2.0.0

BlueVoiceLink1 2.0.0 - Central

osxBlueVoice 2.0.0

Source code reference application based on

osxBlueVoice 2.0.0

NUCLEO-F401RE

NUCLEO-L476RG

Microphone

Bluetooth Low Energy

X-NUCLEO-CCA02M1

X-NUCLEO-IDB04A1

STM32 Nucleo

18

ST BlueMS + BlueVoiceLink1Simplex voice over BLE communication from STM32 ODE to Android/iOS

19

Cloud-based

ASR Service

In configuration

8 kHzADPCM

Available on Google Play

and App Store

NUCLEO-F401RE

NUCLEO-L476RG

NUCLEO-L053R8

Microphone

BLE

X-NUCLEO-CCA02M1

X-NUCLEO-IDB04A1

STM32 Nucleo

BlueVoiceLink1 2.0.0 - Peripheral

osxBlueVoice 2.0.0

ST BlueMS + Bluemicrosystem2BlueVoice Service Integration in BlueMicrosystem environment

Cloud-based

ASR Service

ST BlueMS available on Google Play and App Store

8 kHzADPCM

20

BlueVoiceLink1 2.0.0 - Peripheral

osxBlueVoice 2.0.0

BlueVoice Applications

ADPCM

@ 8 kHzADPCM

@8 or 16 kHz

ST BlueMS

3.0.0

BlueMicrosystem2.1.0 BlueVoiceLink1 2.0.0

Peripheral

BlueVoiceLink1 2.0.0

Central

data rates ≤ 32kbps

to maximize

compatibility

From version 4.4 KITKAT

API 19

R&D

From version 8

NUCLEO F401

NUCLEO L476

SensorTile

NUCLEO F401

NUCLEO L476

NUCLEO L053

BlueCoin

osxBlueVoice 2.0.0 osxBlueVoice 2.0.0

osxBlueVoice 2.0.0

NUCLEO F401

NUCLEO L476

Cloud-based

ASR Service

21

BlueVoice Advanced Application

SensorTileIntegrated Development Platform

Core System

Cradle eXpansion

Cradle

23

SensorTileSoftware platform

24

Starter Projects BlueMicrosystem BlueVoice Function Pack

Core System

SensorTile Native Support for BlueVoice 25

SensorTile

Terminal

Platform Signals

Comm

Interface

Cloud-based

Services

Mobile

Device«Natural Language»

BlueVoice over BlueNRG-1BLE Application Processor

26

Audio Sampling frequency: 8 kHz

Compression algorithm: ADPCM

Bitrate 32kbps

Receiver can be any STM32

BlueVoice client or an Android/iOS

device running ST BlueMS app

STEVAL-IDB007V1

BlueNRG-1 evaluation platform

Demo application:

BlueVoice R&D 27

R&D

R&D

BlueVoice

FullDuplex Speech

BlueVoice

FullBand Music

BlueVoice R&D – FullBand Music 28

Compression algorithm: Opus• Highly-flexible speech and audio codec

• Bit-rates from 6 kb/s to 510 kb/s

• Narrowband (8 kHz) to fullband (48 kHz)

• Frame sizes from 2.5 ms to 60 ms

• Speech and Music support

• Mono and Stereo

• Flexible rate control

• Flexible complexity

• All changeable dynamically

• Completely free• Royalty-free licensing

• Open-source implementation

• IETF RFC 6716

Stereo

Full Band Music @ up to 48 kHz

Full Band Audio

Streaming

USB

I2S

Stereo

Notifications

BlueVoice Control Data

Central/Client - RX

USB /I2S

16bit@48 kHz

Compressed

AudioPCM

BLE

48kbps

PCMCompressed

Audio

BLE

Peripheral/Server - TX

16bit@48kHz

USB

.wav48kbps

Full band

music @ 48kbps Notifications Write without resp

Full Band Audio Streaming

• BlueNRG-MS

• STM32F446, STM32F7

• Audio Format: PCM @ 48 kHz stereo

• Opus Compression @ 48kbps

BlueVoice FullBand Demo Overview 29

BlueVoice R&D – FullDuplex Speech 30

Bidirectional (Full-Duplex)

audio streaming

Input: PCM @ 16 kHz

Opus Compression @ 10 kbps

or higher (Variable Bit Rate)

BlueVoice R&D – FullDuplex Speech

Central Data Packet Peripheral Data Packet

Radio

Idle

t

…

Connection

Event

Connection

Event

t

……Radio

Idle

Connection

Event

Connection

Event

Connection

Event

Connection

Event

Central GATT Interval

Peripheral GATT Interval

Different Central-Peripheral interleaving schemas may be implemented

31

BlueVoice Solutions

Half-duplex High quality musicFull-duplex Full band music

OpusADPCM

32/64 kbps

8 - 16 kbps

SBC

<64 kbps

<64 kbps >64 kbps

8 - 16 kbps

32/64 kbps

32 - 64 kbps

8 - 10 kbps

Compression schema

R&D

Demo

R&D

Demo

Simplex

R&D

32/64 kbps Demo

<64 kbps

R&D

32/64 kbps

32/64 kbps

R&D

Demo

R&D

32

BlueVoice Solutions

Advanced

features

Ultra-Low Power

BlueVoice-L

• STM32L4 or BlueNRG-1

• Features

• ADPCM @8kHz

• Hw PCM filtering

• Low power mic

BlueVoice+

• Advanced audio

processing

• Beamforming

• ASR

BlueVoiceLink

• STM32F4/STM32L4/STM32L0

• Features

• Half Duplex Speech

• ADPCM @16/8kHz

• ST BlueMS app Android/iOS

Basic

functonalities

High Performance

BlueVoice Full Band

• Opus Compression

• High Quality Stereo

Music

• 48+ KHz sampling rate

BlueVoice-Full Duplex

• Full Duplex Speech

• Opus @ 16 kHz

• Low Latency

BlueVoice-Android

• Half Duplex Speech

• ADPCM @ 8/16kHz

• SBC/Opus @ 16 kHz

33

Thank You