artificial intelligence - web.uvic.caweb.uvic.ca/~afyshe/talks/ccn_ai.pdfartificial intelligence...
TRANSCRIPT
![Page 1: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/1.jpg)
Artificial Intelligencewith connections to neuroimaging
1
Slides are available at:
http://web.uvic.ca/~afyshe/talks/ccn_AI.pdf
Alona [email protected]
http://web.uvic.ca/~afyshe/http://brainyblog.net
![Page 2: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/2.jpg)
What is AI?
2
How would we know if we had achieved AI?
![Page 3: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/3.jpg)
Robot Student Test (Goertzel, 2012)
• If a computer could…
– Register in a university program
– Attend classes
– Read the textbook
– Successfully complete tests/assignments
– Finish a degree
… that would be artificial intelligence
3
![Page 4: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/4.jpg)
4
![Page 5: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/5.jpg)
AI2’s Aristo: 8th Grade Science Tests
• Allen Institute for AI
– Training an AI to “read” science textbooks and the internet
– Take 8th grade science tests
http://allenai.org/aristo/5
![Page 6: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/6.jpg)
AI2’s Aristo: 8th Grade Science Tests
http://allenai.org/aristo/6
![Page 7: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/7.jpg)
AI2’s Aristo: 8th Grade Science Tests
http://allenai.org/aristo/7
![Page 8: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/8.jpg)
AI2’s Aristo: 8th Grade Science Tests
http://allenai.org/aristo/8
![Page 9: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/9.jpg)
AI2’s Aristo: 8th Grade Science Tests
9http://allenai.org/aristo/
![Page 10: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/10.jpg)
Robot Student Test (Goertzel)
• What would that require?
– perception
• e.g. read figures in books
– represent knowledge
• e.g. foxes are like wolves
– communication via language (written at least)
• e.g. writing essays for class
– learn, reason, generalize, plan…
10
![Page 11: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/11.jpg)
Robot Student Test (Goertzel)
• What would that require?
– perception
• e.g. read figures in books
– represent knowledge
• e.g. foxes are like wolves
– communication via language (written at least)
• e.g. writing essays for class
– learn, reason, generalize, plan…
11
![Page 12: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/12.jpg)
This talk is not a complete overview
• There are many, many areas of AI I am not covering today
– Reinforcement Learning
– All of robotics (navigation, planning, interaction with the physical world…)
– …
12
![Page 13: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/13.jpg)
The Singularity
• I will not talk about AI taking over the world
• Instead, I will talk about
– how AI currently impacts society
– how biased models sneak their way into our lives
13
![Page 14: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/14.jpg)
SKILL 1: PERCEPTION
The Robot Student Test
14
![Page 15: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/15.jpg)
Perception in AI
• Amazing advances in computer vision
– ImageNet
• First released 2009
• 1.2 million images
• more than 1000 concepts – e.g. cup, oil filter, ptarmigan
– Deep learning
• CNNs
http://image-net.org/ 15
![Page 16: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/16.jpg)
ImageNet
16http://cs.stanford.edu/people/karpathy/cnnembed/
![Page 17: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/17.jpg)
ImageNet
17http://image-net.org/challenges/LSVRC/2010/browse-synsets
![Page 18: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/18.jpg)
18
http://image-net.org/challenges/LSVRC/2012/analysis/
![Page 19: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/19.jpg)
https://blogs.nvidia.com/blog/2016/01/12/accelerating-ai-artificial-intelligence-gpus/19
Krizhevsky, Sutskev & HintonConvolutional Neural Net (CNN)
![Page 20: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/20.jpg)
20
Self Driving Cars
![Page 21: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/21.jpg)
Self Driving Cars
21
![Page 22: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/22.jpg)
Neural Nets: Crash course
22http://cs231n.github.io/neural-networks-1/
![Page 23: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/23.jpg)
Neural Nets: Crash course
23
x1
x2
x3
x4
x5
j2
j3
h1
h2
h3
y1
y2
y3
Input (e.g. an image)
Output (e.g. predict object
in image)
o
![Page 24: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/24.jpg)
Neural Nets: Crash course
24
x1
x2
x3
x4
x5
j2
j3
h1
h2
h3
y1
y2
y3
Input (e.g. an image)
Output (e.g. predict object
in image)
o
![Page 25: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/25.jpg)
Neural Nets: Crash course
• Architecture fixed (e.g. depth, # neurons)
• Weights at each edge are learned
– Backpropagation (stochastic gradient descent)
25
![Page 26: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/26.jpg)
Convolutional Neural Nets
26
Two additional operations: • Convolution• Pooling/Subsampling
![Page 27: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/27.jpg)
Convolutional Neural Nets (CNNS)
27https://upload.wikimedia.org/wikipedia/commons/6/63/Typical_cnn.png
![Page 28: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/28.jpg)
CNNs: Convolution
28
https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0
1 0 1
0 1 0
1 0 1
Input image (here binary, RGB typical)
Filter (here binary, but typically continuous values)
![Page 29: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/29.jpg)
CNNs: Convolution
29
https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0
1 0 1
0 1 0
1 0 1
Convolve, shift, convolve
![Page 30: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/30.jpg)
CNNs: Convolution
• Typical Filters ( )
30Krizhevsky et al.
1 0 1
0 1 0
1 0 1Filters are learned
![Page 31: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/31.jpg)
Feature Map
.
.
.
CNNs: Convolution
http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/ (Fergus intro)
• Output of convolution layer is a feature map
( )4 3 4
2 4 3
2 3 4
![Page 32: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/32.jpg)
CNNs: Pool/Subsample
32Krizhevsky et al.
4 3 4
2 4 3
2 3 4
Average = 3.2
Max = 4
![Page 33: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/33.jpg)
CNNs: Pool/Subsample
33Krizhevsky et al.
4 3 4
2 4 3
2 3 4
Average = 3.2
Max = 4
• Supplies some invariance to local translations• Reduces the dimension of subsequent layers
![Page 34: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/34.jpg)
CNNs: Hidden representations
• Activations (feature maps) at each layer are a “hidden representation” of the image
• A compression of the information in image
– not unlike PCA/SVD
34
![Page 35: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/35.jpg)
What do the neurons represent?
35
x1
x2
x3
x4
x5
j2
j3
h1
h2
h3
y1
y2
y3
o
How can we make j2’s valueas high as possible?
What stimuli makes j2fire maximally?
![Page 36: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/36.jpg)
36Horikawa & Kamitani (2017)
![Page 37: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/37.jpg)
37Horikawa & Kamitani (2017)
![Page 38: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/38.jpg)
What do CNNs have to do with the brain?
• You keep using the word neuron…
38
![Page 39: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/39.jpg)
39
![Page 40: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/40.jpg)
40
Horikawa & Kamitani (2017)
Horikawa & Kamitani (2017)
![Page 41: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/41.jpg)
41
…
…
8 CNN layers
7 vision-relatedareas
Caveat: fMRI data
x1
x2
x3
x4
x5
j2
j3
h1
h2
h3
y1
y2
y3
o
![Page 42: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/42.jpg)
CNNs vs the Brain
• Which predictions are most correlated with the true CNN representations?
42
…
…
8 CNN layers
7 vision-relatedareas
Train all possible 56 models to predict CNN
hidden representations from fMRI data
![Page 43: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/43.jpg)
43Horikawa & Kamitani (2017)
Co
rrel
atio
n c
oef
fici
ent
![Page 44: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/44.jpg)
44Horikawa & Kamitani (2017)
Co
rrel
atio
n c
oef
fici
ent
![Page 45: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/45.jpg)
CNNs vs the Brain
• There is a relationship between higher vision brain areas and higher levels of the CNN
46
![Page 46: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/46.jpg)
SKILL 2: REPRESENTING KNOWLEDGE
The Robot Student Test
47
![Page 47: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/47.jpg)
How can we represent knowledge?
48
![Page 48: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/48.jpg)
How can we represent knowledge?
49
Sweetness
Grows on a tree
orangeapple
car
banana
sandwich
![Page 49: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/49.jpg)
Vector Space Models (VSMs) of Semantics
50
![Page 50: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/50.jpg)
Vector Space Models of Semantics
51
(0,0.03)
Sweetness
Grows on a tree
carsandwich
orangeapplebanana
![Page 51: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/51.jpg)
Scaling it up
• How can we differentiate >10k English words?
– more dimensions (many more!)
• Need to create dimensions and assign values automatically
![Page 52: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/52.jpg)
Vector Space Models of Semantics
• Every word gets a vector
Apple
Banana 1 .9 9 0 .3 -1 -3
1 .8 .3 4 -.2 2 1
53
Orange 1 .7 .9 2 -.4 1 .1
…
…
…
![Page 53: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/53.jpg)
Representing Semantics
• Process a large text corpus
• Find words associated with word of interest
– banana appears
• often with verb eat
• less often with verb drive
• Latent Semantic Analysis (LSA)(Landauer and Dumais, 1997)
• SkipGram (sometimes called Word2Vec)(Mikolov et al. 2013)
54
![Page 54: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/54.jpg)
SkipGram
• Another neural network (surprise!)
• Given a central word (e.g. banana), predict probable context words (e.g. ate, yellow)
• Use a corpus to generate pairs of central and context words
55
Joe ate the banana, which was yellow
![Page 55: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/55.jpg)
56
x1
x2
x3
x4
xV
h1
h2
h3
y1
y2
Inputone-hot vector
(1 only for the central word, 0 elsewhere)
Output probability distribution
over all words
…
y4
yV
…Train so that y values are highfor commonlyco-occurring words, low for other words
Hidden representation for
central word
x2
y3y3
y4
![Page 56: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/56.jpg)
SkipGram
• We call the hidden representation for each central word a word vector
• These vectors have (seemingly) magical properties*
57*Actually, it’s the data itself that’s magical. See Levy, Goldberg, & Dagan (2015)
h1
h2
h3
![Page 57: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/57.jpg)
What can word vectors do?
• Solve analogies
– Hammer is to nail as screwdriver is to…?
• screw
– Solve by finding word closest to
(nail – hammer + screwdriver)
58
![Page 58: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/58.jpg)
Linguistic Regularities
59Mikolov, Sutskever, Chen, Corrado, & Dean (2013)
![Page 59: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/59.jpg)
Linguistic Regularities
60Mikolov, Yih & Zweig (2013)
![Page 60: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/60.jpg)
61Mikolov, Sutskever, Chen, Corrado, & Dean (2013)
![Page 61: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/61.jpg)
What can word vectors do?
• Predict word similarity
62
wo
rd2
vec
dim
1
word2vec dim 2
orangeapple
car
banana
SkipGram space
orangeapple
car
banana
human rating space
![Page 62: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/62.jpg)
63
1 SkipGram layer
4 fMRI vectors(frontal, temporal, occipital, parietal)
x1
x2
x3
x4
xV
h1
h2
h3
y1
y2
…
y4
yV
…
y3
cat
![Page 63: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/63.jpg)
RSA-style analysis, SkipGram <-> fMRI
64Ruan, Ling & Hu (2016)
![Page 64: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/64.jpg)
BrainBench
The brain as a test bed for
learned word representations
Xu, Murphy & Fyshe, 2016
65
![Page 65: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/65.jpg)
BrainBench
• Tests your favorite word vectors against the brain’s representations
• Multiple Modalities– fMRI
– MEG
– EEG
• English and Italian
• Abstract and concrete nouns
• And growing!!
Xu, Murphy & Fyshe, 2016
66
![Page 66: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/66.jpg)
Average across 4 datasets
Non-distributional
Xu, Murphy & Fyshe, 2016
http://www.langlearnlab.cs.uvic.ca/brainbench/
Haoyan (Ed)Xu
DhanushDharmaretnam
67
![Page 67: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/67.jpg)
SkipGram vs the Brain
• SkipGram vectors are correlated with fMRI activity
• So are lots of other word vector models
68
![Page 68: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/68.jpg)
SKILL 3: COMMUNICATE VIA LANGUAGE
The Robot Student Test
69
![Page 69: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/69.jpg)
Recurrent Neural Network
• Neural networks that write
• Predict the next word as a function of
– context
– previous
• Write that word, and recurse
70
![Page 70: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/70.jpg)
x1
x2
x3
x4
h1
h2
h3
y1
y2
y4
yV
…
y3
InputWord vector for the
current word
Output probability distribution
over next word in sequence
c1
c2
c3
CONTEXT Context? Use the hidden representation from previous generation step
Sample from this distribution to determine the next word
![Page 71: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/71.jpg)
Recurrent Neural Network Language Model
INPUT (t)
CONTEXT (t-1)
CONTEXT (t) OUTPUT (t+1)
apple
zoo
…
Harry
…
fly
…
broom
…
aardvark
…
aEMBEDDING (t)
Mikolov, Kombrink, Deoras, Burget & Cernocky (2011)
![Page 72: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/72.jpg)
Recurrent Neural Network Language Model
INPUT (t)
CONTEXT (t-1)
CONTEXT (t) OUTPUT(t+1)
apple
zoo
…
Harry
…
fly
…
broom
…
aardvark
…
a
had
had
never
never
believed
believed
Harry
Harry
Wehbe et al. 2014
![Page 73: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/73.jpg)
Sunspring
74https://arstechnica.com/gaming/2016/06/an-ai-wrote-this-movie-and-its-strangely-moving/
![Page 74: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/74.jpg)
What do RNNs have to dowith the brain?
75
![Page 75: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/75.jpg)
Harry Potter and the Reading Brain
• MEG
• Read Chapter 9 of Harry Potter and the Sorcerer’s Stone.
• Read one word at a time, each word for 0.5 s
Wehbe et al. 2014
![Page 76: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/76.jpg)
When and where do the brain processes occur?
Conjecture
(a) Story context before seeing word w
(b) Perception of word w
(c) Integration of word w
word w+1word w-1 word w
0.5 s0.5 s 0.5 s
a
b
c
Leila WehbeWehbe et al. 2014
![Page 77: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/77.jpg)
Recurrent Neural Network Language Model
INPUT (t)
CONTEXT (t-1)
CONTEXT (t) OUTPUT (t+1)
apple
zoo
…
Harry
…
fly
…
broom
…
aardvark
…
aEMBEDDING (t)
Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and J Cernocky.RNNLM- recurrent neural network language modeling toolkit.ASRU Workshop 2014.
Wehbe et al. 2014
![Page 78: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/78.jpg)
Parallelism
Brain processes:
(a) Represent context
(b) Retrieves properties
(c) Integrate the word with context
Neural Net components:
(a) Context vector
(b) Word Embedding
(c) P(w | context)
Wehbe et al. 2014
![Page 79: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/79.jpg)
Annotate every word with new features
Embedding of word w
Context (before word w
is seen)
Probability of word w
given context
Word w
Word w
Wehbe et al. 2014
![Page 80: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/80.jpg)
Recurrent Neural Network Language Model
• Learned on Harry Potter fan fiction database. (60 million words)
– Nearest Neighbors of Harry:
• James, Jinny, Lilly, Albus, Ron
• Model then “reads” Chapter 9 of The Sorcerer’s Stone word by word, and produces vectors of interest.
Wehbe et al. 2014
![Page 81: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/81.jpg)
Embedding of word w
Context (before word w
is seen)
Probability of word w
given context
Word w
Word w
Wehbe et al. 2014
![Page 82: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/82.jpg)
Results: Context vector by time
Wehbe et al. 2014
![Page 83: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/83.jpg)
Back
Left Right
visual
frontal
temporal
Results: time/sensor Context before word w
Embedding of word w
Probability of word w
Wehbe et al. 2014
![Page 84: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/84.jpg)
RNNs vs the Brain
• There is a relationship to the representations an RNN learns and information in the brain
• Caveat: is this actually what reading is?
85
![Page 85: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/85.jpg)
Robot Student Test (Goertzel)
• What would that require?
– perception
• e.g. read figures in books
– represent knowledge
• e.g. foxes are like wolves
– communication via language (written at least)
• e.g. writing essays for class
– learn, reason, generalize, plan…
86
![Page 86: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/86.jpg)
AI = Neural Networks?
• Of course not…
87
![Page 87: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/87.jpg)
Bias in AI
88
![Page 88: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/88.jpg)
89
![Page 89: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/89.jpg)
Bias in AI
• Recall word vectors can solve analogy problems:
• e.g. Paris : France as Tokyo : xWord vectors will tell you that x = Japan.
• father : doctor as mother : xx = nurse
• man : computer programmer as woman : xx = homemaker
90https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/
![Page 90: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/90.jpg)
Bias in, Bias out?
• People have biases… should models have bias?
• Biases are harmful, even deadly, if perpetuated
– Face recognition works better for white people
• Increased mistaken identity among people of color
– Software used to decide parole/sentencing had elevated risk assessments for people of color that did not correlate to actual recidivism rates
91https://www.theatlantic.com/technology/archive/2016/04/the-underlying-bias-of-facial-recognition-systems/476991/https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
![Page 91: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/91.jpg)
Bias in, Bias out?
• Our models may even exaggerate bias!
• “For example, the activity cooking is over 33% more likely to involve females than males in a training set, and a trained model further amplifies the disparity to 68% at test time.”
92Zhao (2017) https://arxiv.org/pdf/1707.09457.pdf
![Page 92: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/92.jpg)
Bias in Artificial Intelligence
• Combating bias in AI is an open research question
yearly conference FATML
http://www.fatml.org/
See also:
https://qz.com/1064035/google-goog-explains-how-artificial-intelligence-becomes-biased-against-women-and-minorities/ (lots of good links at the bottom)
https://cdt.org/issue/privacy-data/digital-decisions/
93
![Page 93: Artificial Intelligence - Web.UVic.caweb.uvic.ca/~afyshe/talks/ccn_AI.pdfArtificial Intelligence with connections to neuroimaging 1 Slides are available at: ... •e.g. read figures](https://reader031.vdocument.in/reader031/viewer/2022022511/5adfb8a87f8b9a97518c5140/html5/thumbnails/93.jpg)
Thanks!Alona Fyshe
[email protected]://web.uvic.ca/~afyshe/
http://brainyblog.net
These slides available at: http://web.uvic.ca/~afyshe/talks/ccn_AI.pdf