Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
Combining Semantics and Deep Learning for Intelligent Information ServicesAntrittsvorlesung
Prof. Dr. Harald SackAIFB, 29.11.2017
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20172
http://www.telegraph.co.uk/science/2017/10/18/alphago-zero-google-deepmind-supercomputer-learns-3000-years/
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20173
The Futile Tries of “Strong” AI20 years of “AI Winter”... https://www.flickr.com/photos/x-ray_delta_one/4128131032
"in from three to eight years we will have a machine with the general intelligence of an average human being", Marvin Minsky (1970)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
Inspired by Biology...
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20175 https://www.flickr.com/photos/x-ray_delta_one/4128131032 https://commons.wikimedia.org/wiki/File:Blausen_0657_MultipolarNeuron.png
From Biological Neuron to the Artificial Neuron Modell - McCulloch & Pitts (1943)
(Dendrites) (Soma) (Axon)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20176
Cognitive Computing - The MARK 1 Perceptron (1957)
http://techgenix.com/tgwordpress/wp-content/uploads/2017/01/perceptron.jpg
weight update
error
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20177
Timeline of Neural Networks
https://www.slideshare.net/deview/251-implementing-deep-learning-using-cu-dnn/4
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20178 https://www.flickr.com/photos/x-ray_delta_one/4128131032
Deep Convolutional Neural Networks on GPU Supercomputers
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.20179
Visual Concept Detection
= the ability to learn visual categories in order to automatically identify
new, unseen images of these categories only based on visual content
https://pixabay.com/p-2175353/https://cloud.google.com/vision/
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201710
Visual Concept Detection as Machine Learning Task
Supervised Learning:
● Positive images (that depict the concept)
● Negative images (that don’t)
● Classification/Prediction:
○ Test image, if it depicts concept (or not): ??
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201711
Size Matters
To achieve high quality results, we need sufficient training data
● Influence of training data size on classification accuracy for
○ Deep Convolutional Neural Networks (CNN)
vs.
○ Aggregated local features and linear predictor (IFV)
● Results:
○ CNNs largely benefit from bigger datasets
○ IFVs are a competitive candidate esp. if only limited
training data is availableC. Hentschel, T. Wiradarma, H. Sack: If we did not have imagenet: Comparison of fisher encodings and convolutional neural networks on limited training data (AVC 2016)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201712
Leveraging Social Media to Improve Visual Content Detection
Extending MIRFLICKR-1M
● 1 Million Flickr images (selection based on interestingness score)
● Additional image metadata (authoritative & user created): text
● How to select appropriate training data?
○ Text: word2vec skip-gram model
to determine related (similar) tags
for a given query
○ Images: visual reranking
to filter images visually similar
to top ranked images
C. Hentschel, H. Sack: Learning from the Uncertain -- Improving Image Classifiers with Community Training Data (i-KNOW 2015)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201713
What Do Classifiers Really See?
● Heatmaps representing the influence of an image region on the classification result
C. Hentschel and H. Sack, What Image Classifiers Really See – Visualizing Bag-of-Visual Words Models (MMM 2015)
Aggregated local features and
linear predictor (IFV)
Deep Convolutional
Neural Networks (CNN)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201714
● Near-human level image classification
● Near-human level speech recognition
● Near-human level handwriting transcription
● Improved machine translation
● Improved text-to-speech conversion
● Digital assistants such as Google Now or Amazon Alexa
● Near-human level autonomous driving
● Superhuman Go playing
What Deep Learning has achieved so far
https://media.wired.com/photos/59268c8ccfe0d93c474309b2/master/pass/GettyImages-627219854.jpg
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
How to Represent Knowledge?
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201716 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg
The Universal Categories - Aristotle (384–322 BC)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201717 https://upload.wikimedia.org/wikipedia/commons/c/ce/Gottfried_Wilhelm_Leibniz%2C_Bernhard_Christoph_Francke.jpg
Calculemus!
Calculus Ratiocinator - Gottfried Wilhelm Leibniz (1646-1716)
„..alle menschlichen Schlussfolgerungen müssten auf irgendeine mit Zeichen arbeitende Rechnungsart zurückgeführt werden, wie es sie in der Algebra und Kombinatorik und mit den Zahlen gibt, wodurch nicht nur mit einer unzweifelhaften Kunst die menschliche Erfindungsgabe gefördert werden könnte, sondern auch viele Streitigkeiten beendet werden könnten, das Sichere vom Unsicheren unterschieden und selbst die Grade der Wahrscheinlichkeiten abgeschätzt werden könnten, da ja der eine der im Disput Streitenden zum anderen sagen könnte: Lasst uns doch nachrechnen!“
Leibniz in a letter to Ph. J. Spener, Juli 1687
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201718
Begriffsschrift - Gottlob Frege (1848-1925)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201719 https://www.flickr.com/photos/x-ray_delta_one/4128131032
Frames for Represent Knowledge - Marvin Minsky (1974)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201720
The Renaissance of “Soft” AICarol Kaelson/Jeopardy Productions Inc., via Associated Press
From Linked Data to Knowledge Graphs
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201721
Knowledge Graphs for Natural Language Processing
rdf:type
dbo:Philosopher
rdfs:subClassOf
dbo:Person
21
Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz.
dbr:Gottfried_Willhelm_Leibniz dbr:Christian_Wolffdbo:doctoralAdvisor
dbo:Philosopher
rdf:type
dbr:Ontology
dbo:notableIdea
text
knowledge base
foaf:name
dbo:birthDate
“Gottfried Wilhelm Leibniz“@de
“1646-07-01”^^xsd:date
“1716-11-14”^^xsd:datedbo:deathDate
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201722
Knowledge Graphs for Natural Language Processing
22
Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz.
text
Named EntitiesCommon Entities
Named Entity Linking Language Model● Statistical Context Analysis (co-occurrence)
Knowledge Graph● Graph Analysis (connected components)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201723
Knowledge Graphs for Natural Language Processing
23
Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz.
1. Create potential entity candidates
2. Filter entity candidates by NER type
3. Create induced subgraph of knowledge graph
4. Determine connected components
N. Steinmetz, H. Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions (ESWC 2013)
J. Waitelonis, H. Sack, Named Entity Linking in #Tweets with KEA, (Microposts 2016)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201724
Knowledge Graphs for Question Answering
“Where was Leibniz born?”
Entity Linking wd:Q9047
(Gottfried Wilhelm Leibniz)
Result Type wd:Q618123
(geographical object)
Relation Extraction wdt:P19
(place of birth)
Natural Language
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201725
Knowledge Graphs for Question Answering
“Where was Leibniz born?”
Entity Linking
Result Type
Relation Extraction wdt:P19
(place of birth)
wd:Q9047
(Gottfried Wilhelm Leibniz)
Natural Language
wd:Q618123
(geographical object)
SPARQL Query
SELECT ?o WHERE { wd:Q9047 wdt:P19 ?o . ?o wdt:P31/wdt:P279* wd:Q618123 .}
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201726
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201727
What Knowledge Graphs have achieved so far
● Improved search results on the web
● Answering natural language questions
● Suggest content-based recommendations
● Enable serependitious discoveries
● Enables exploratory search
● Large scale data integration
● Still missing: common sense knowledge
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
How to combine Deep Learning and Semantics?
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201729
▶ Deep Learning for Knowledge Graphs ▶
● NLP and Knowledge Extraction via Deep Learning to populate and extend
Knowledge Graphs
● NLP and Knowledge Extraction via Deep Learning for Ontology Learning to
extend and refine Knowledge Graphs
● NLP and Graph Analysis supported by Deep Learning for Ontology
Alignment and Link Discovery to combine and integrate Knowledge
Graphs
◀ Knowledge Graphs for Deep Learning ? ◀
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201730
Word Embeddings (word2vec, glove)
Leibniz wrote to Caroline of Ansbach that Newton’s physics was detrimental to natural theology. However, eager to defend the Newtonian view, it was Clarke who responded and the correspondence between both continued until the death of Leibniz.
0.2860.792
…−0.177−0.100.109
−0.542…
0.3490.271
● Words are represented as vectors that preserve the linguistic context
● Semantically similar words are represented in close neighborhood within the vector space
● Enable analogies via vector arithmetics
T. Mikolov et al., Efficient Estimation of Word Representations in Vector Space, archivx 2013
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201731
Knowledge Graph Embeddings (rdf2vec)
0.2860.792
…−0.177−0.100.109
−0.542…
0.3490.271
● RDF graph are represented as vectors that preserve the semantic context
● Semantically similar entities are represented in close neighborhood within the vector space
● Enable analogies via vector arithmetics
dbr:Gottfried_Wilhelm_Leibniz
dbr:Erhard_Weigel
dbr:Gottlob_Frege
dbr:Hanover
dbr:Leipzig
dbc:German_Mathematician
dbo:academicAdvisor
dbo:influenced
dct:subjectdct:subject
dct:subject
dbr:University_of_Leipzig
dbo:almaMater
dbo:city
dbo:birthPlace
dbo:deathPlace
P. Ristovski et al., RDF2Vec: RDF Graph Embeddings and Their Applications, SWJ 2016
M. Cochez et al, Global RDF Vector Space Embeddings, ISWC 2017
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201732
Combined Feature Embeddings for a Compound Knowledge Space
● Various feature vectors
○ Word embeddings
○ Knowledge Graph embeddings
■ Instances■ Ontologies
○ Embeddings for semantically enriched texts
○ Metadata and aggregated features
0.2710.123-0.24
-0.2860.792
…−0.177−0.100.109
−0.542…
0.3490.56
-0.1320.1130.91
...0.560.99
0.2710.123-0.24
-0.334
Text space
Knowledge space
Context space
CompoundKnowledge Space
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201733
Towards Neuro-Symbolic Integration
Neuro-Symbolic Systems
1. Translation of symbolic (background)
knowledge into the network
2. Learning of additional knowledge
from examples (and generalisation) by
the network
3. Executing the network
(i.e. reasoning), and
4. Symbolic knowledge extraction
from the network.
Network ensembles
Levels of abstraction
Besold et al.: Neural-Symbolic Learning and Reasoning: A Survey and Interpretation (2017)
specialization
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
Deep Learning and Semantics for Information Services
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201735T. Tietz, J. Waitelonis, J. Jäger, H. Sack, refer: a Linked Data based Text Annotation and Recommender System for Wordpress, (ISWC 2016)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.201736
From Information Retrieval to Information ExplorationT. Tietz, J. Jäger, J. Waitelonis, H. Sack, Semantic Annotation and Information Visualization for Blogposts with refer, (VOILA 2016)
Combining Semantics and Deep Learning for Intelligent Information Services, Prof. Dr. Harald Sack, AIFB Inaugural Lecture, 29.11.2017
Prof. Dr. Harald SackFIZ Karlsruhe, Leibniz Institute for Information Infrastructure
AIFB, KIT Karlsruhe
Combining Semantics and Deep Learning for Intelligent Information Services