음성 ui - 봇(bot)과사물인터넷의 설계및개발 · 2019-06-27 · title: how to use the...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted

음성 UI - 봇(Bot)과사물인터넷의설계및개발Voice User Interface & Assistant & Bots & IoT

황주필 (jupil.hwang@oracle.com)유원조 (wonjo.yoo@oracle.com)

Sales Consultant, Infrastructure CloudAug 30, 2017

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Confidential – Oracle Internal/Restricted/Highly Restricted

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface의변화CLI / GUI ??

Confidential – Oracle Internal/Restricted/Highly Restricted 3

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

HAL 9000

Confidential – Oracle Internal/Restricted/Highly Restricted 4

Computer

HAL & Computer

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface

Confidential – Oracle Internal/Restricted/Highly Restricted 5

?

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface

Confidential – Oracle Internal/Restricted/Highly Restricted 6

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface – CLI

Confidential – Oracle Internal/Restricted/Highly Restricted 7

CLI - Command Line Interface

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface

Confidential – Oracle Internal/Restricted/Highly Restricted 8

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface – GUI (마우스에서손가락으로)

Confidential – Oracle Internal/Restricted/Highly Restricted 9

GUI – Graphical User Interface

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Evolution of interaction

Confidential – Oracle Internal/Restricted/Highly Restricted 10

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

User Interface - Voice

Confidential – Oracle Internal/Restricted/Highly Restricted 11

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 12

Mobile Voice Interface

2011 2012 2014

Speech Recognition에서 AI로의발전

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Voice User Interface - Wars

스마트폰에서 Apple Siri와 Google Now같은음성인식 AI비서서비스로경쟁을했지만, Amazon이 2014년인공지능스피커 Echo를출시, 음성인식소프트웨어인 Alexa의개발자키트를외부에오픈 Skill이빠르게늘어남 (검색, 쇼핑, 뱅킹등등..) Echo 판매량증가

Confidential – Oracle Internal/Restricted/Highly Restricted 13

기업명 내용

Apple - 2010년 ‘Siri’ (음성인식소프트웨어) 인수- 2016년 VocalIQ (음성인터페이스) 인수

Amazon - 2011년 ‘Yap’ (STT : Speech-to-Text) 인수- 2014년 ‘Evi’ (지식기반 Semantic Search Engine), ‘INOVA’ (TTS : Text-to-Speech) 인수

Facebook - 2013년 ‘Jibbigo’ (모바일다국어음성번역) 인수- 2015년 ‘Wit.at’ (음성인식), ‘Two Big Ears’ (VR 오디오) 인수

Google - 2014년 ‘DeepMind technologies’ (AI 머신러닝) 인수- 2016년 API.AI (음성인식, 자연어처리) 인수, Limes Audio(음성처리) 인수

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Voice User Interface - Wars

Confidential – Oracle Internal/Restricted/Highly Restricted 14

회사 이름 API / SDK

Amazon Alexa (2014) Alexa voice service sdk 제공, 한국어지원안함

Google Assistant (2016) Google assistant sdk 제공, 한국어지원

Apple Siri (2011) Siri SDK for iOS, macOS 제공, 한국어지원

SKT Nugu (2016) X

KT GigaGeenie (2017) 제휴신청필요 (법인심사 2주소요)

Naver Wave (2017) 음성인식 ‘Clova’ API 공개(Android iOS SDK), 일 3600초까지무료

Kakao Kakao mini (2017) 3분기 AI앱/스피커출시예정, 음성인식 “뉴톤” Android,iOS SDK

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Voice User Interface - Wars

• Amazon – Alexa Voice Service (AVS)

• 2015 – Alexa Skills Kit (ASK) 공개, 2017년 1분기이용가능한 Skills 개수가 10,000 개 (Uber호출, Amazon물건구입, 스타벅스커피주문등..)

Confidential – Oracle Internal/Restricted/Highly Restricted 15

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

나만의음성인식비서를만들어보자

Confidential – Oracle Internal/Restricted/Highly Restricted 16

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

DIY Alexa !

Confidential – Oracle Internal/Restricted/Highly Restricted 17

집에 Raspberry Pi 하나쯤은있잖아요 !

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

DIY Alexa !

Confidential – Oracle Internal/Restricted/Highly Restricted 18

Step 1 … 10 따라하기

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

DIY Alexa !

Alexa-avs-sample-app

http://github.com/alexa/alexa-avs-sample-app

Alexa 개발자 – AVS 활성화 – 기기등록

Alexa! How is the weather today??

Confidential – Oracle Internal/Restricted/Highly Restricted 19

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Google Assistant 는?

Confidential – Oracle Internal/Restricted/Highly Restricted 20

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

• google-oauthlib-tool –client-serect/home/pi/client_id.json –scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless

• googlesampeles-assistant-hotword

Confidential – Oracle Internal/Restricted/Highly Restricted 21

DIY Google Home !

• Raspberry Pi – Audio Enable

• Audio Lib 설치

• Python Module 설치 with pip– pip install –upgrade google-assistant-

sdk

– pip install –upgrade google-auth-oauthlib

• Google APIs > credentials > client id 등록및 client-secret json다운로드후 oAuth tool을통해서credentials.json 생성

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

한번보시죠Demo

Confidential – Oracle Internal/Restricted/Highly Restricted 22

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Home Automation with Google Assistant

Confidential – Oracle Internal/Restricted/Highly Restricted 23

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

AI Speaker - 아키텍처

AI. Speaker Cloud

How’s the weather?

기본기능시간,날씨,메모,음악

Intent 파악

응답 Speech 생성

인터넷검색

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

AI Speaker - 아키텍처

AI. Speaker Cloud

How’s the weather?

Intent 파악

종료

Custom 기능Alexa Skill, Google Action

Other 시스템

“Ask my skill” “Talk to my Home”

Webhook

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Home Automation - 아키텍처

음성인식,출력 Case에따른 Device 호출 Device 제어

자연어인식

의도(intent) 파악

Custom 기능 Action 호출

기기(TV, 에어컨, 조명)

명령 (On, Off)

기기제어 API 호출

기기제어 API

적외선(IR) 호출

Python Simple Http

Server

AI Speaker SDK

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Home Automation - 아키텍처

Confidential – Oracle Internal/Restricted/Highly Restricted 27

Google Assistant SDK

Google Actions

Google Api.AI

Python Simple Http

Server

BroadLinkRM3 Mini

Python API

ngrokWebhook

https

HomeNetwork

Server

TCP

IR (적외선)

IRCustomAction

Talk to“My Action”

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

ngrok 이란?

이미지출처 https://ngrok.com/

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

TV, 에어컨제어 - IR (적외선)

Confidential – Oracle Internal/Restricted/Highly Restricted 29

직접만들수있음Receiver, Sender, LIRC단점: 공구비용, 단방향

BroadLink RM3 Mini

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

TV, 에어컨제어 - IR (적외선)

Confidential – Oracle Internal/Restricted/Highly Restricted 30

TV

Audio

Aircon

사용자추가기기

192.168.1.111192.168.1.100

Send SIGNAL Code

Learn SIGNAL

IR Signal

TV ON Code : 26001399401923….Audio ON Code : 14454512345451….Aircon ON Code : 6234973491-384….

TCP

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

TV, 에어컨제어 - Code

Confidential – Oracle Internal/Restricted/Highly Restricted 31

pi@raspberrypi:~/rm3/python-broadlink/cli $ ls

README.md aircon.turnoff broadlink_discovery tv.channel1 tv.channel5 tv.channel9 tv.turnon

aircon.downtemp aircon.turnon MyHomeServer.py tv.channel2 tv.channel6 tv.channeldown tv.volumedown

aircon.morewind aircon.uptemp device.info tv.channel3 tv.channel7 tv.channelup tv.volumeup

aircon.smallwind broadlink_cli tv.channel0 tv.channel4 tv.channel8 tv.turnoff

pi@raspberrypi:~/rm3/python-broadlink/cli $ ./broadlink_cli -h

usage: broadlink_cli [--device DEVICE] [--type TYPE] [--host HOST] [--send SEND] [--sensors] [--learn] [--learnfile

LEARNFILE]

--send SEND send command

--learnfile LEARNFILE learn command and save to specified file

pi@raspberrypi:~/rm3/python-broadlink/cli $ cat tv.turnon

2600139940192398109823874098123098120398120974812738129739128673129038012983012938216498123213

BroadlinkRM3 Mini

Python API

01340891801…….

34923043123……

tv.channelup

tv.turnon

broadlink_cli

https://github.com/mjg59/python-broadlink

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

TV, 에어컨제어 - Code

Confidential – Oracle Internal/Restricted/Highly Restricted 32

if command == 'channel':

if level != '':

if level == '11':

exeCmd = './broadlink_cli --send @' + device + '.' + command + "1"

subprocess.call(shlex.split(exeCmd))

exeCmd = './broadlink_cli --send @' + device + '.' + command + "1"

subprocess.call(shlex.split(exeCmd))

elif command == 'volumedown' or command == 'volumeup':

for i in range(1,numlevel):

exeCmd = './broadlink_cli --send @' + device + '.' + command

subprocess.call(shlex.split(exeCmd))

...

...

def run(server_class=HTTPServer, handler_class=S, port=9876):

server_address = ('', port)

httpd = server_class(server_address, handler_class)

print("Starting httpd... 9876")

httpd.serve_forever()

Python Simple Http

Server

BroadLinkRM3 Mini

Python API

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Oracle Bot Integration with Alexa

Confidential – Oracle Internal/Restricted/Highly Restricted 33

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Live Demo - Pizza 주문

Confidential – Oracle Internal/Restricted/Highly Restricted 34

Oracle's Intelligent Bot

피자주문

크러스트

사이즈

피자종류

사이드

User결제수단

기존카드정보확인

접속요청

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Oracle Bot Integration with Alexa - 아키텍처

Confidential – Oracle Internal/Restricted/Highly Restricted 35

AmazonAVS SDK

Alexa Voice Service

Alexa Skill(Oracle Pizza)

ask“Oracle Pizza”

Oracle Intelligent Bot

Channel

Channel Configurator

DialogFlow

Execution

CustomComponents

AI EngineWelcome to

oracle pizza

Cloud

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 36

Alexa Skill 설정

{"intents": [

{"slots": [

{"name": "command","type": "LITERAL"

}],"intent": "CommandBot"

},{

"intent": "AMAZON.StopIntent"}

]}

Intent schemaSkill invocation name

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 37

Oracle Intelligent Bot 의 Alexa Channel 설정

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 38

Oracle Intelligent Bot 동작방식

“I want to order a pizza” “Can I order large thin cheese pizza?”

메시지수신

1

의미파악

2

처리를위한특정정보기억

3

답변수행

4

Intent

Utterance

Entity

Dialog Flow

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 39

Oracle Intelligent Bot 의 Intent, Entity 설정

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Web기반음성인식 with Oracle Bot

Confidential – Oracle Internal/Restricted/Highly Restricted 40

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Web기반음성인식 with Oracle Intelligent Bot

Confidential – Oracle Internal/Restricted/Highly Restricted 41

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 42

Live Demo - 한국어음성인식 Oracle Bot Demo

Oracle's Intelligent BotUser Chrome 브라우저

Backend System

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Web기반음성인식 with Oracle Intelligent Bot - Code

Confidential – Oracle Internal/Restricted/Highly Restricted 43

if (!('webkitSpeechRecognition' in window)) {

upgrade();

} else {

start_button.style.display = 'inline-block';

recognition = new webkitSpeechRecognition();

...

recognition.onresult = function(event) {

var interim_transcript = '';

for (var i = event.resultIndex; i < event.results.length; ++i) {

if (event.results[i].isFinal) {

var recData = event.results[i][0].transcript;

sendToBot(recData);

...

var u = new SpeechSynthesisUtterance();

speechSynthesis.speak(u);

Ajax로 Oracle Bot 호출

음성인식객체

음성출력

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Summary

Confidential – Oracle Internal/Restricted/Highly Restricted 44

• Machine Learning 의발달로점점완벽한음성인식가능

• AI Speaker 시장의확대

•다양한음성인식플랫폼의전성시대

• ChatBot 음성인식 Bot으로의전환

•음성인식시스템의 SDK의오픈개발자들의다양한기회

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Let’s go home to Code!!

Confidential – Oracle Internal/Restricted/Highly Restricted 45

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Confidential – Oracle Internal/Restricted/Highly Restricted 46