Download - [2C3]Developing context-aware applications
![Page 2: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/2.jpg)
Anticipatory computing is transforming the way we find information
Today, we find information Tomorrow, information finds us
Set reminder
- Launch calendar app - Create new reminder - Enter flight details
Check flight status
- Launch web browser - Go to airline site - Enter flight number
Check traffic
- Launch web browser - Go to map / traffic site - Enter current location - Enter airport address
![Page 3: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/3.jpg)
Anticipatory computing relies on context awareness S
ourc
e: “
Ent
oura
ge”
by H
BO
![Page 4: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/4.jpg)
Sou
rce:
Our
Mob
ile P
lane
t by
Goo
gle
Smartphone adoption (2013)
![Page 5: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/5.jpg)
Mobile devices capture context via many sensors
Sou
rce:
Sam
sung
Sou
rce:
funf
.org
- Cameras - Microphones - Cellular receiver - Wi-Fi receiver - GPS receiver - Gyroscope - Thermometer - Barometer - …
![Page 6: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/6.jpg)
Backend systems infer user situation, activity, intent, mood S
ourc
es: G
igaO
M, R
obin
Lab
s
![Page 7: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/7.jpg)
Responsive design
Source: Mashable
![Page 8: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/8.jpg)
Contextual design
![Page 9: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/9.jpg)
Recent technology advances
Speech Recognition - Deep learning (deep / recurrent ANNs) - Ultra large language models - Dynamic speaker adaptation - Massive datasets (108s of users)
Computer Vision - Deep learning - Massive datasets
Language Understanding - Deep learning - Knowledge graphs
Sou
rce:
Fac
eboo
k S
ourc
e: S
tanf
ord
Uni
vers
ity
![Page 10: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/10.jpg)
Knowledge graphs From disembodied strings to grounded entities
• Yahoo! 10 M entities, 30 M properties, 10 M connections • Microsoft 300 M entities, 800 M connections • Google 570 M entities, 18 B properties and connections • Wikipedia 4 M entities • Freebase 40 M topics, 2 B facts • Factual 66 M local businesses and POIs in 50 countries • LinkedIn 225 M people • Facebook 1.15 B people Cf. • Cyc
239 K concepts, 2 M facts
• OpenCyc 6 K concepts, 60 K facts
Sou
rce:
Yah
oo
![Page 11: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/11.jpg)
Dynamic activation of the knowledge graph TIME
Continuous user context
hayes valley palo alto north beach cow hollow
I really want to see that new movie with Ben Affleck It is the one about the Iran Hostage Crisis
I am going to meet Raymond at Goat Hill Pizza at noon
You have to see that video of the Today Show doing the Harlem Shake It is near the Comstock Saloon
The Black Keys were on the Colbert Report last night I am planning to go whitewater rafting in the Grand Canyon
It is near the Comstock Saloon
Rolling Context Window
Dynamic entity graph (~10M entities)
things I recently wrote or said
restaurants near North Beach
places in the Bay area
topics related to things I recently read
current events my friends, colleagues
and recent contacts links that my friends have recently shared
Human Knowledge (~50B entities)
5B people
1B places 1B products
100M interests
100M events 1B media
2008 (1M entities)
2010 (10M entities)
2014 (500M entities)
2016 (10B entities)
5B domain-specific
![Page 12: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/12.jpg)
The knowledge graph enables anchored NLP
“I saw the man on the hill with the telescope”
Sou
rce:
Den
iz Y
uret
![Page 13: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/13.jpg)
Voice
10% of Baidu search queries are done with voice today. In five years, it’ll be 50% ”
Andrew Ng
![Page 14: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/14.jpg)
Types of voice-driven applications
Question & Answer "What is the capital of California?" "Who directed Citizen Kane?"
Command & Control "Call Jenny's work phone." "Turn up the heat to 72 degrees."
Content Discovery "Is there a good Japanese restaurant near Union Square?" "Show me all the James Bond movies with Roger Moore."
Performing Tasks "Make a reservation for two at Kama tomorrow at 8pm." "Book me on a flight to JFK on Saturday afternoon."
Dictation "Send a text to Jenny saying…" "Send the following email to Joe…"
Passive Listening "…have you seen that video of the Russian meteor…" "…I’m thinking of getting a pair of red Kobe 9 sneakers…"
![Page 15: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/15.jpg)
Anatomy of a voice interaction
1. Speech recognition 2. Natural language understanding
3. Search ranking & filtering 4. Real-time visualization of results
type: restaurant category: Italian
location: San Francisco cost: $, $$
filter: good for kids
”It’d be nice to find an inexpensive Italian restaurant in San Francisco that is good for kids.”
Candidate 1: Buca di Beppo [confidence: 0.91] Candidate 2: La Traviata [confidence: 0.82]
Candidate 3: Ragazza [confidence: 0.80] Candidate 4: Sotto Mare [confidence: 0.76]
…
![Page 16: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/16.jpg)
The MindMeld platform
generate a continuously changing model of user intent based on long-running context 2�1�
passively analyze multiple concurrent data streams for each user in real-time voice, gps, video, updates, … 3�
proactively find, correlate and rank relevant information display to user as appropriate
![Page 17: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/17.jpg)
The MindMeld API
![Page 18: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/18.jpg)
CONFIDENTIAL
Step 1 !We will automatically index !any document collection.!
18
![Page 19: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/19.jpg)
![Page 20: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/20.jpg)
CONFIDENTIAL
Step 1 !We will automatically index !any document collection.!!!Step 2 !Use our API to continuously!track contextual signals for!your users.!!
20
![Page 21: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/21.jpg)
curl -‐X POST \ -‐H "X-‐MindMeld-‐Access-‐Token: mindmeld-‐access-‐token" \ -‐H "Content-‐Type: application/json" \ -‐d '{ "text": "I was thinking we could go to Muir Woods or Stinson Beach", "type":"speech", "weight":0.5 }' \ "https://mindmeld.expectlabs.com/session/:sessionid/textentries"
21
![Page 22: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/22.jpg)
CONFIDENTIAL
Step 1 !We will automatically index !any document collection.!!!Step 2 !Use our API to continuously!track contextual signals for!your users.!!!Step 3 !Display context-driven !search results and !recommendations.!!!
![Page 23: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/23.jpg)
curl "https://mindmeld.expectlabs.com/session/:sessionid/documents"
![Page 24: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/24.jpg)
Speech transcription
Syntactic analysis
Keyphrase & entity
extraction
Utterance classification
& conversation
modeling
Correlation & extrapolation
Doc ranking
Utterance propagation
![Page 25: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/25.jpg)
Technology choice: Speech recognition
Google Nuance
AT&T Build own
• Google’s server-side implementation of HTML5’s webkitSpeechRecognition for Chrome and Android
• Fast, interim results, free • 79 languages • No SLA, no iOS support
• Nuance NDEV Mobile speech-to-text service • Custom vocabularies • Embedded engine, cloud-based API, and
combination • 40 languages • SLA
• AT&T Speech API • iOS, Android SDK clients to cloud-based API • 19 languages • SLA
• Start with open-source speech recognition engine such as Sphinx or Kaldi
• Be ready to invest $$$ • Full control
![Page 26: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/26.jpg)
Technology choice: Natural language understanding
Stanford NLP NLTK
AlchemyAPI TextRazor
• PoS tagging, parsing, entity extraction, co-reference resolution
• Java-based SDK • Free (GNU General Public License v2)
• PoS tagging, parsing, entity extraction • Python-based SDK • Free (Apache License v2)
• Sentiment analysis, entity / keyword extraction, language detection
• Cloud-based API
OpenCalais Build own
• Extraction of named entities, facts, and events • Cloud-based API
• Combine, extend functionality • Full control • Requires maintenance
• Entity recognition, topic tagging, dependency parsing
• Cloud-based API
![Page 27: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/27.jpg)
Technology choice: Machine learning
Scikit-learn Weka
Google GraphLab
• Classification, regression, clustering • SVM, logistic regression, random forests, … • Python-based, open-source library
• Data analysis, predictive modeling • Wide array of machine learning classifiers • Java-based, free library (GNU GPL)
• Google Prediction API • Pattern matching, classifiers, recommender
systems • Freemium pricing
PredictionIO Build own
• Predictive modeling, recommender systems • Open-source service
• Combine, extend functionality • Optimize runtime for each model
• Topic modeling, graph analytics, clustering, collaborative filtering, computer vision
• Parallel programming • C++ core, Python interface
![Page 28: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/28.jpg)
Technology choice: Development & operations
GitHub Chef
Nginx Nagios
• Code repos management
• Operations monitoring & alerting
Circle CI Pivotal Tracker
• Project management
• Server provisioning
• Web server
• Test & build
Amazon Web Services Nitrous.IO
• Cloud-based servers, storage, load balancers • Dev box in seconds with browser-based IDE
![Page 29: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/29.jpg)
Challenges
Functionality Accuracy
Scalability • Users • Applications • Domains Latency • ASR • NLU • IR • Visualization
ASR • Word error rate • Accented speech • Noisy environment • Distant speaker NLU / IR • Precision & recall • Word sense disambiguation • Anaphora resolution • Conversation modeling • Interruptability
![Page 30: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/30.jpg)
MindMeld API: Powerful yet easy to use
developer.expectlabs.com real-time location
on any device speech recognition on any device
Android SDK iOS
SDK
JavaScript SDK
sample code
turnkey HTML5 widgets
push events
open graph support
entity extraction
customizable ranking
keyphrase detection
topic detection
natural language processing
on-demand web crawling
proactive suggestions
instant answers
extensive online documentation
real-time analytics console
complete API explorer tool
crawl manager dashboard
ranking dashboard
![Page 31: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/31.jpg)
MindMeld API: Adaptive ranking factors
![Page 32: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/32.jpg)
MindMeld API: Powering a wide range of applications
Voice-Driven Intelligent Assistant
Location-Based Proactive Assistant
Voice and Video Conference Assistant
1 2 3
online commerce media & entertainment mobile apps & devices
wearables
location-based services local & travel apps
smart cars mobile workforce
customer support & help desk call center solutions voice & video calling
telepresence & collaboration
![Page 33: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/33.jpg)
Instant voice-driven search and discovery on your own content
deve
lope
r.ex
pect
labs
.com
![Page 34: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/34.jpg)
감사합니다
![Page 35: [2C3]Developing context-aware applications](https://reader033.vdocument.in/reader033/viewer/2022051609/547e88345806b5b85e8b467f/html5/thumbnails/35.jpg)
Widescreen Test Pa.ern (16:9)
Aspect Ra8o Test
(Should appear circular)
16:9
4:3