computers have feelings too
TRANSCRIPT
![Page 2: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/2.jpg)
And you are?Paul Glavich
Principal Consultant @ readify
Author
Former MVP of 13 years
International Speaker (if you count NZ)
Industry veteran > 20 years
![Page 3: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/3.jpg)
AgendaCognitive services – Overview
Setting up
API Examination, Analytics, Emotion, Vision
Flailing while demo fails
Future?
Wrap up
![Page 4: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/4.jpg)
Microsoft AI Overview
Cognitive Services
• Prebuilt/Pre-trained Models
• Dev / Human friendly
Machine learning,
Bots
• Cortana Intelligence Suite
• Bot framework
• Build/train your own models
Cognitive toolkit
• Deep learning
• Neural networks
• Script driven
![Page 5: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/5.jpg)
Cognitive Services OverviewMicrosoft Cognitive Services - an API platform
That’s right – It is all about the “Apey Eyes”
Approx. 30 Artificial intelligence API’s
![Page 6: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/6.jpg)
Current ListVision
◦ Computer Vision API, Emotion API PREVIEW, Video Indexer PREVIEW, Face API, Video API PREVIEW, Content moderator, Custom Vision Service PREVIEW
Speech◦ Custom Speech Service PREVIEW, Translator Speech API, Bing Speech API, Speaker Recognition API PREVIEW
Language◦ Language Understanding Intelligent Service PREVIEW, Bing Spell Check API, Web Language Model API PREVIEW, Text Analytics API
PREVIEW, Translator Text API, Linguistic Analysis API PREVIEW
Knowledge◦ Recommendations API PREVIEW, Academic Knowledge API PREVIEW, Knowledge Exploration Service PREVIEW, QnA Maker API
PREVIEW, Entity Linking Intelligence Service API PREVIEW, Custom Decision Service PREVIEW
Search◦ Bing Autosuggest API, Bing Video Search API, Bing Image Search API, Bing Web Search API, Bing News Search API, Bing Custom Search
Labs◦ Project Prague, Gesture-based controls, Project Johannesburg, Route logistics, Project Abu Dhabi, Distance matrix, Project Nanjing,
Isochrones calculations, Project Cuzco, Event associated with Wikipedia entries, Project Wollongong
![Page 7: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/7.jpg)
Current ListVision
◦ Computer Vision API, Emotion API PREVIEW, Video Indexer PREVIEW, Face API, Video API PREVIEW, Content moderator, Custom Vision Service PREVIEW
Speech◦ Custom Speech Service PREVIEW, Translator Speech API, Bing Speech API, Speaker Recognition API PREVIEW
Language
◦ Language Understanding Intelligent Service PREVIEW, Bing Spell Check API, Web Language Model API PREVIEW, Text Analytics API PREVIEW, Translator Text API, Linguistic Analysis API PREVIEW
Knowledge◦ Recommendations API PREVIEW, Academic Knowledge API PREVIEW, Knowledge Exploration Service PREVIEW, QnA Maker API PREVIEW, Entity Linking
Intelligence Service API PREVIEW, Custom Decision Service PREVIEW
Search◦ Bing Autosuggest API, Bing Video Search API, Bing Image Search API, Bing Web Search API, Bing News Search API, Bing Custom Search
Labs◦ Project Prague, Gesture-based controls, Project Johannesburg, Route logistics, Project Abu Dhabi, Distance matrix, Project Nanjing, Isochrones calculations,
Project Cuzco, Event associated with Wikipedia entries, Project Wollongong
![Page 8: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/8.jpg)
The favourite childLUIS
◦ Language Understanding Intelligent Service
Conversational API (Apey Eye )
https://azure.microsoft.com/en-us/services/cognitive-services/language-understanding-intelligent-service/
We wont be looking at this
![Page 9: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/9.jpg)
What we will be looking atTextAnalytic API
◦ Extract information such as sentiment, key phrases, topics and language from your text
Emotion API◦ Examines facial expression in an image as an input, and returns the confidence across a set of emotions
for each face in the image, as well as bounding box for the face.
![Page 10: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/10.jpg)
But before we dive inWhat about some of the other players?
Google◦ Machine Learning, Cloud Natural Language API, Cloud Speech API, Cloud Translation API, Cloud Vision API,
Cloud Video Intelligence API BETA◦ Tensorflow, Kaggle, API.AI◦ Deepmind – acquired in 2014
AWS◦ Lex, Polly, and Rekognition, Alexa voice service◦ Speech recognition, natural language understanding, lifelike speech, image recognition, and image analysis
Salesforce?◦ Yep◦ Can distinguish customers from imposters by tracking action patterns
And others….
![Page 11: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/11.jpg)
SetupAdd Cognitive services to your azure subscription
◦ Manually
◦ ARM Template
◦ Powershell
Access keys
Location
![Page 12: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/12.jpg)
Rest Call formathttps://[location].api.cognitive.microsoft.com/[service]/[version]/[action]
Headers required◦ Content-type
◦ Ocp-Apim-Subscription-Key
![Page 13: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/13.jpg)
SentimentSimple Json structure to submit text documents
ID used to correlate response
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment
Request with synchronous response
Supports English, Spanish, French, and Portuguese with 12 more in preview.
![Page 14: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/14.jpg)
DemoTextAnalytic Raw API usage
Is that positive or negative?
![Page 15: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/15.jpg)
Key phrasesSame Json structure as Sentiment
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases
Request with synchronous response
Currently supports English, German, Spanish and Japanese.
![Page 16: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/16.jpg)
DemoKey phrases Raw API usage
We can determine whether it is positive or negative, but what are the key phrases in that?
![Page 17: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/17.jpg)
LanguageSimilar Json structure as Sentiment
Request with synchronous response
120 Languages are supported
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages
![Page 18: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/18.jpg)
DemoLanguage Raw API usage
Watch your language
![Page 19: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/19.jpg)
TopicsSame Json structure as Sentiment
Stop words – excluded from topic detection
Stop phrases – excluded from returned results
Minimum 100 Documents
Submit request, then check on operation status.
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/languages
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/operations/{operationId}
![Page 20: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/20.jpg)
Topics responseInitial submit
◦ 202 Accepted
◦ “operation-Location” header also returned
◦ https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/operations/54a25882d7ef494ca779b31d46f842bb
OperationStatus
![Page 21: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/21.jpg)
EmotionCan detect
◦ Happiness
◦ Sadness
◦ Surprise
◦ Anger
◦ Fear
◦ Contempt
◦ Disgust
◦ Neutral
Works with video as well (via async operation)
https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize
![Page 22: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/22.jpg)
DemoEmotion API Raw API usage
![Page 23: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/23.jpg)
VisionCategorise content of an image
Find faces
Determine if contains mature/adult content
Url or image upload
https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=...&details=...&language=...
Supports English as default and Simplified Chinese as language to return.
![Page 24: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/24.jpg)
VisionSpecify what you want as query string parameters
visualFeatures◦ Categories - categorizes according to taxonomy defined in documentation.
◦ Tags – returns list of words related to the image content.
◦ Description - describes image with a complete English sentence.
◦ Faces – if faces are present, return coordinates, gender and age.
◦ ImageType - clipart or line drawing.
◦ Color - accent color, dominant color, or black&white.
◦ Adult – Adult in nature? Racy in nature?
details◦ Celebrities - identifies celebrities if detected in the image.
◦ Landmarks - identifies landmarks if detected in the image.
![Page 25: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/25.jpg)
DemoVision API Raw Usage
![Page 26: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/26.jpg)
A Fluent APIBest way to learn (IMHO) is do.
Fluent API on top of cognitive services
1. TextAnalytics
2. Emotion
3. Vision and more to come
![Page 27: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/27.jpg)
Simple usage
![Page 28: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/28.jpg)
DemoSentiment and keyphrase analysis of emails
Display sentiment result of emails.
http://dddsydney2017glavsdemo.azurewebsites.net
![Page 30: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/30.jpg)
Demo - high level
Gmail
Scheduler Q
Collect Email
Text Analysis
CosmosDB
Website
Send
Collect
Analyse
Store
Analyse
Get Mail
Analysedresults
Analysed Email
Azure Sentiment
Analysis
![Page 31: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/31.jpg)
Other fluent examplesShow me the tests
Fluent API
◦ https://github.com/glav/CognitiveServicesFluentApi
![Page 32: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/32.jpg)
What I think the future holdsMore consumerisation of AI
Specialised services beyond “simple API’s”
More compute “on the device” with greater value augmented by the cloud
Google’s API.AI is a step towards easier consumption
More seamless application integration.
![Page 34: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/34.jpg)
What now?Visual Studio
◦ File -> New -> Console Application (.Net core)
Install-Package Glav.CognitiveServices.FluentApi.TextAnalytic –Pre
var result = awaitTextAnalyticConfigurationSettings.CreateUsingConfigurationKeys(TestConfig.TextAnalyticsApiKey, LocationKeyIdentifier.WestUs)
.SetDiagnosticLoggingLevel(LoggingLevel.Everything)
.AddDebugDiagnosticLogging()
.UsingHttpCommunication()
.WithTextAnalyticAnalysisActions()
.AddSentimentAnalysis("I am having a fantastic time.")
.AnalyseAllSentimentsAsync();
![Page 35: Computers have feelings too](https://reader031.vdocument.in/reader031/viewer/2022022415/5a66e82a7f8b9a91298b48cf/html5/thumbnails/35.jpg)
Links and resourcesThe demo
◦ https://github.com/glav/DDDSydney2017SentimentDemo
Fluent API
◦ https://github.com/glav/CognitiveServicesFluentApi
Nuget Packages
◦ https://www.nuget.org/packages/Glav.CognitiveServices.FluentApi.TextAnalytic/0.5.0-alpha
◦ https://www.nuget.org/packages/Glav.CognitiveServices.FluentApi.Emotion/0.5.0-alpha