oscon 2008: mashing up voice and the web using open source and xml

July 23, 2008

OSCON 2008

Dan YorkDir. of Emerging Comm [email protected]

Mashing up Voice and the Web Through Open Source and XML

Why Voice?

The Problem

Telephony Sucks

Complex

Photo: http://www.flickr.com/photos/28481088@N00/

Arcane

Proprietary

Simple

Ubiquitous

The Solution?

Web Developers

Phone Developers Web Developers

They Understand XML

Let's build an API...

Proprietary = Lock-In

Standards Matter

Lock-In

The Building Blocks

Building Blocks

VoiceXML

SIP

CCXML

Application

Platform

Network

XML+HTTP

SIP

Session Initiation Protocol

- Session Initiation Protocol (SIP)defines how to establish a communication session betweentwo endpoints

- Primarily used for voice, but can for IM or virtually any other protocol

- Almost always used in client/server configuration with "SIP proxies" in control of "SIP endpoints"- Work going on in P2PSIP - see www.p2psip.org

- Text-based protocol, originally modeled on HTTP

SIP Communication

Alice Bob

RTP (voice)

INVITE

180 RINGING

200 OK

ACK

BYE

200 OK

Alice Bob

ProxyA

ProxyBSIP

SIP

RTP

SIP

SIP Resources

- Internet Engineering Task Force (IETF)- RFC 3261- Hitchhiker’s Guide to SIP

- Open Source Info- VoIP Info Wiki: www.voip-info.org

- Industry Sites- SIP Forum: www.sipforum.org- SIP Foundry: www.sipfoundry.org- OpenSBC: www.opensourcesip.org

Open Source SIP Software

- Systems- Asterisk: www.asterisk.org- sipXecs: www.sipfoundry.org- FreeSWITCH: www.freeswitch.org- OpenSER: www.openser.org

- SIP Stacks- reSIProcate: www.resiprocate.org

- Phones:- Gizmo: www.gizmoproject.org

- MANY, many more: www.voip-info.org

Application

Platform

Network

XML+HTTP

SIP

So how does this work?

Web Browser Diagram

WebBrowser

WebSvr

HTTP

HTML

AppSvr?

PHPperl python

Java???

ruby

XMLXML

servlets

Web Browser Diagram - Thin Client

WebBrowser(on svr)

WebSvr

HTTP

HTML

AppSvr?

PHPperl python

Java???

ruby

XMLXML

servlets

ThinClient Protocol

Voice Browser Diagram

VoiceBrowser(on svr)

WebSvr

HTTP

XML

AppSvr?

PHPperl python

Java???

ruby

XMLXML

servlets

Phone Audio

VoiceXML

- W3C standard to define speech dialogs

- Defines prompts and grammars that together create a phone application

- Thousands of developers

- 30+ Open source projects

- Can include JavaScript

- http://www.w3.org/TR/voicexml21/

VoiceXML Hello World<?xml version="1.0" encoding="UTF-8"?><vxml version = "2.1" >

<form> <block> <prompt> Hello World. This is my first telephone application. </prompt> </block> </form>

</vxml>

VoiceXML Elementsassignaudioblockbreakcatchchoicecleardatadisconnectelseelseifemphasisenumerateerrorexampleexitfieldfilledforeachformgotogrammarhelpifinitialitemlinklogmark

menumetanoinputnomatchone-ofoptionparagraphparamphonemepromptpropertyrecordrepromptreturnrulerulerefsay-asscriptsendsentencesubsubdialogsubmittagthrowtokentransfervaluevar

CCXML

- Call Control XML (CCXML) is the W3C standard for call control using XML

- Sister standard to VoiceXML

- Integrates with VoiceXML for dialog control

- Provides a framework for issuing call control commands and handling call control events

- http://www.w3.org/TR/ccxml/

CCXML and State

Connected DisconnectedAlerting

(Ringing)

Events trigger transitions between states.

CCXML Hello World<?xml version="1.0" encoding="UTF-8"?><ccxml version="1.0">

<eventprocessor>

<transition event="connection.alerting"> <log expr="'***** CONNECTION ALERTING *****'"/> <if cond="event$.connection.remote == '8315551234'"> <reject/> <else/> <accept/> </if> </transition>

<transition event="connection.connected"> <log expr="'***** CALL WAS ANSWERED *****'"/>

</transition>

<transition event=”connection.disconnected”> <log expr=”‘*** Call was disconnected ***’”/> <exit/> </transition>

<transition event="error.*"> <log expr="'an error has occured (' + event$.reason + ')'"/> <exit/> </transition>

</eventprocessor></ccxml>

CCXML Elementsacceptassignauthenticatecancelcreatecallcreateccxmlcreateconferencedestroyconferencedialogpreparedialogstartdialogterminatedisconnectelseelseifeventprocessorexitfetch

foreachgotoifjoinlogmetametadatamoveredirectrejectscriptsendtransitionunjoinvar

ENOUGH ALREADY...

SHOW ME THE CODE!

Application

Platform

Network

XML+HTTP

SIP

For this demo, I'm using

evolution.voxeo.combut it could be any

VXML platform

Demo #1Seeking to solve the universal question...

IS DOWN?

Demo #1 - Is Twitter Down?

- VoiceXML & JavaScript

- Connects to www.istwitterdown.com

- Uses JavaScript to parse the result

- Relays result to caller using Text-To-Speech

- Demo:- 1-206-701-7091- sip:[email protected]

Demo #1: VoiceXML & JavaScript <?xml version="1.0" encoding="UTF-8"?>

<vxml version = "2.1">

<var name="MyData"/>

<form id="F1">

<script> <![CDATA[ function GetData(d,t,n) { return (d.getElementsByTagName(t).item(n).firstChild.data);} ]]> </script>

<block> <data name="MyData" src="http://www.istwitterdown.com/"/> <assign name="document.MyData" expr="MyData.documentElement"/>

<assign name="status" expr="GetData(MyData,'a',0)"/> <if cond="status=='No'"> <prompt>Twitter is currently up. Yea!</prompt> <else/> <prompt>Twitter is currently down. </prompt> </if> </block> </form></vxml>

Demo #2Listen to identi.ca

Demo #2 - Listen to identi.ca

- VoiceXML

- Reads out latest identi.ca notices

- Caller says:- "friends"- "replies"- "public"

- Caveat- hardcoded to single identi.ca user

- Demo- 1-617-401-7088- sip:[email protected]

Demo #2: VoiceXML<?xml version="1.0" encoding="UTF-8"?><vxml version = "2.1">


<form id="F1"> <script> <![CDATA[ function GetData(d,t,n) { return (d.getElementsByTagName(t).item(n).firstChild.data);} ]]> </script> <field name="Choice"> <audio src="../audio/identicachoice.wav"/>

<grammar xmlns="http://www.w3.org/2001/06/grammar" xml:lang="en-US" root="MYRULE"> <rule id="MYRULE"> <one-of> <item>friends</item> <item>replies</item> <item>public</item> </one-of> </rule> </grammar>

<noinput>I did not hear anything. Please try again. <reprompt/> </noinput> <nomatch> I did not recognize that word. Please try again. <reprompt/> </nomatch> </field> <filled namelist="Choice"> <if cond="Choice == 'friends'"> <data name="MyData" src="http://identi.ca/danyork/all/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>Your last notice is from <value expr="GetData(MyData,'dc:creator',0)"/>. The notice is: <value expr="GetData(MyData,'title',2)"/>. </prompt> <elseif cond="Choice == 'replies'"/ <data name="MyData" src="http://identi.ca/danyork/replies/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>Your last reply is from <value expr="GetData(MyData,'dc:creator',0)"/>. The reply is: <value expr="GetData(MyData,'title',2)"/>.</prompt> <elseif cond="Choice == 'public'"/> <data name="MyData" src="http://identi.ca/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>The last public notice is from <value expr="GetData(MyData,'dc:creator',0)"/>. The notice is: <value expr="GetData(MyData,'title',1)"/>. </prompt> </if> <prompt> That is all. Thank you for calling in. </prompt> </filled> </form></vxml>

Demo #2: VoiceXML<?xml version="1.0" encoding="UTF-8"?><vxml version = "2.1">


<form id="F1"> <script> <![CDATA[ function GetData(d,t,n) { return (d.getElementsByTagName(t).item(n).firstChild.data);} ]]> </script> <field name="Choice"> <audio src="../audio/identicachoice.wav"/>

<grammar xmlns="http://www.w3.org/2001/06/grammar" xml:lang="en-US" root="MYRULE"> <rule id="MYRULE"> <one-of> <item>friends</item> <item>replies</item> <item>public</item> </one-of> </rule> </grammar>

<noinput>I did not hear anything. Please try again. <reprompt/> </noinput> <nomatch> I did not recognize that word. Please try again. <reprompt/> </nomatch> </field> <filled namelist="Choice"> <if cond="Choice == 'friends'"> <data name="MyData" src="http://identi.ca/danyork/all/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>Your last notice is from <value expr="GetData(MyData,'dc:creator',0)"/>. The notice is: <value expr="GetData(MyData,'title',2)"/>. </prompt> <elseif cond="Choice == 'replies'"/ <data name="MyData" src="http://identi.ca/danyork/replies/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>Your last reply is from <value expr="GetData(MyData,'dc:creator',0)"/>. The reply is: <value expr="GetData(MyData,'title',2)"/>.</prompt> <elseif cond="Choice == 'public'"/> <data name="MyData" src="http://identi.ca/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>The last public notice is from <value expr="GetData(MyData,'dc:creator',0)"/>. The notice is: <value expr="GetData(MyData,'title',1)"/>. </prompt> </if> <prompt> That is all. Thank you for calling in. </prompt> </filled> </form></vxml>

if cond="Choice == 'friends'"> <data name="MyData" src="http://identi.ca/danyork/all/rss?limit=1"/> <assign name="document.MyData" expr="MyData.documentElement"/> <prompt>Your last notice is from <value expr="GetData(MyData,'dc:creator',0)"/>. The notice is: <value expr="GetData(MyData,'title',2)"/>. </prompt>

Demo #3 -Telephony Presence & identi.ca

Demo #3 - Phone Presence & identi.ca

- CCXML

- Sends phone presence (on phone, off phone) to identi.ca

- Caller dials in to the application- Caller is connected to the other number- Results of the call status are posted to identi.ca via their implementation of the Twitter API


<?xml version="1.0" encoding="UTF-8"?><ccxml xmlns="http://www.w3.org/2002/09/ccxml" version="1.0"> <var name="state" expr="'init'"/> <var name="incomingcall"/> <var name="tURL" expr="'http://danyork:[email protected]/apis/statuses/update.xml'"/> <eventprocessor statevariable="state"> <transition event="connection.alerting" state="init"> <accept/> </transition> <transition event="connection.connected" state="init"> <assign name="state" expr="'calling'"/> <assign name="incomingcall" expr="event$.connectionid"/> <createcall dest="'tel:+14079678424'"/> </transition> <transition event="connection.connected" state="calling"> <assign name="state" expr="'connected'"/> <join id1="event$.connectionid" id2="incomingcall"/> <var name="status" expr="'Dan is on the phone'"/> <send targettype="'basichttp'" name="'update'" target="tURL" namelist="status"/> </transition> <transition event="connection.failed" state="calling"> <assign name="state" expr="'done'"/> <var name="status" expr="'Dan is not answering his phone'"/> <send targettype="'basichttp'" name="'update'" target="tURL" namelist="status"/> </transition> <transition event="connection.disconnected" state="connected"> <assign name="state" expr="'done'"/> <var name="status" expr="'Dan is off the phone'"/> <send targettype="'basichttp'" name="'update'" target="tURL" namelist="status"/> </transition> <transition event="send.successful" state="done"> <exit/> </transition> </eventprocessor></ccxml>

Telephony Presence & identi.ca

http://www.w3.org/2002/09/ccxml




<?xml version="1.0" encoding="UTF-8"?><ccxml version="1.0" xmlns="http://www.w3.org/2002/09/ccxml"></ccxml>



Demo #4Audio posts to identi.ca

Demo #4: Audio posts to identi.ca

- Application flow:- User calls in and leaves message- Message is uploaded to Amazon S3- Resulting URL posted to user's identi.ca account

- Caveats:- Single user- Message overwritten each time- No security for S3- Kludgey way of dealing with S3 response


Demo #4: VoiceXML code<?xml version="1.0" encoding="UTF-8"?>

<vxml version = "2.1"><var name="status"/><var name="response"/><var name="S1"/><var name="key"/><var name="acl" expr="'public-read'"/>

<form>

<record name="file" beep="true" dtmfterm="true" type="audio/wav"> <prompt bargein="false"> Please leave your identica audio post after the tone. When you are done, you may press any key. </prompt>

</record>

<catch event="error.badfetch"> <assign name="status" expr="'New audio post at http://oscon.s3.amazonaws.com/message.wav'"/>

<data name="response" namelist="status" method="post" fetchtimeout="3s" src="http://dandemo:[email protected]/api/statuses/update.xml"/>

<prompt>Thank you. Goodbye.</prompt> </catch>

<block>

<assign name="key" expr="'message.wav'"/>

<data name="S1" namelist="key acl file" enctype="multipart/form-data" src="http://oscon.s3.amazonaws.com/" method="post" fetchtimeout="5s"/>

</block>

</form>

</vxml>

<?xml version="1.0" encoding="UTF-8"?>


<var name="status"/><var name="response"/><var name="S1"/><var name="key"/><var name="acl" expr="'public-read'"/>

<form>


</record>




<block>



</block>

</form>

</vxml>

Demo #4: VoiceXML code


</record>

Demo #4: VoiceXML code<?xml version="1.0" encoding="UTF-8"?>



<form>


</record>




<block>



</block>

</form>

</vxml>

More Ideas...

Click-to-call

Project Green Phone

- By Mark Headd

- http://www.voiceingov.org/blog/?p=135

- VoiceXML, JavaScript, CCXML, PHP

- Application:- Caller dials in to app- App uses ANI/Caller ID and makes web service query to a database to determine location

- Makes another web service query to find location of nearest E85/Biodiesel stations

- Relays information to caller- Sends caller a SMS text msg with info

Rocketsource.org

- www.rocketsource.org

- Vox-Attendant

- A VoiceXML-based, speech-driven auto attendant that can connect callers with any person or group with an enterprise, via their desk, cellular, or VoIP phones.

- Vox-Mail

- A VoiceXML-based, speech-driven voicemail application that can store messages locally or integrate with any IMAP email server to provide basic unified messaging.

- Voice Conference Manager

- A VoiceXML and CCXML-based, speech-driven conference manager that features both phone and web-based conference call creation, access, and management.

- VoiceXML, CCXML, grXML, JavaScript, Java, and Python

VoiceXML/CCXML Resources

- Open Source- VoiceXML Platforms

- VoiceGlue: www.voiceglue.org

- Applications- RocketSource: www.rocketsource.org- Festival (TTS): www.cstr.ed.ac.uk/projects/festival/- Sphinx (speech rec): sourceforge.net/projects/cmusphinx/

- Commercial- VoiceXML Platforms

- Voxeo: evolution.voxeo.com

- Information- VoiceXML Forum: www.voicexml.org- W3C: www.w3c.org

Inspiration

Photo: http://www.flickr.com/photos/carbonnyc/

Thank you!

Dan [email protected]

blogs.voxeo.comwww.voxeo.com/freeevolution.voxeo.com

identi.ca/danyork

http://www.voxeo.com/prophecy








oscon 2008: mashing up voice and the web using open source and xml

Technology

open source

xmldan york

emerging comm tech

complex photo

telephony sucks