topological data analysis to understand convolutional...

106
Topological Data Analysis to understand Convolutional Neural Networks Aleksei Prokopev, SeoulAI, 2018

Upload: others

Post on 03-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Topological Data Analysis to understand

Convolutional Neural NetworksAleksei Prokopev, SeoulAI, 2018

Page 2: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Shape

Page 3: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 4: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Topology

Page 5: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 6: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 7: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 8: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Topological Data Analysis

Page 9: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 10: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 11: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 12: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 13: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 14: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Examples

Page 15: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 16: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 17: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Nodes are groups of similar data points

Edges connect similar nodes

Colors let you see values of interest

Position of a node on the screen doesn’t matter

Page 18: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 19: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 20: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 21: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 22: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 23: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 24: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 25: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Convolutional Neural Networks

Page 26: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 27: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 28: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Convolution

Page 29: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 30: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 31: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 32: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 33: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Weights

Page 34: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 35: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 36: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 37: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 38: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 39: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Problems

Page 40: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 41: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 42: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 43: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 44: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

TDA for CNN

Page 45: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 46: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 47: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 48: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 49: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 50: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 51: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 52: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 53: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 54: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 55: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 56: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 57: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 58: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 59: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 60: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 61: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 62: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 63: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 64: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 65: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 66: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Thank you !

Page 67: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 68: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

?

Page 69: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Weapon of choice

Page 70: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 71: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 72: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 73: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 74: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 75: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 76: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 77: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 78: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 79: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

???

Page 80: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 81: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 82: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 83: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 84: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 85: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

HOW MANY NODES?

Page 86: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Ok

Page 87: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 88: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 89: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 90: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 91: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

MEMORY ERROR!!!

Page 92: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Does it make sense?

Page 93: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 94: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 95: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 96: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 97: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 98: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and
Page 99: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Conclusion

Page 100: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Challenges

- Mapper is confusing, too many parameters to tune

- Computations are very memory extensive- Requires sophisticated preprocessing- Toolkits are not perfect

Page 101: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Still very promising!

Page 102: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Why TDA?

- No good understanding what is happening inside Neural Networks, despite of abundance of good research done by very smart people

- Intellectually satisfying and intuitive- Terra incognita

Page 103: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Questions to ask

- How topology changes over layers?- How topology changes over training?- Do different nets have the same underlying

structures?- What do this structures mean?- ...

Page 104: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Further research- CNNs:

- Do they have the same structure?- What happens when overfit?- How topology of learned weights depends on topology of training data- ...

- RNNs:

- What do the cycles mean?

Page 105: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Thank you !

Page 106: Topological Data Analysis to understand Convolutional ...seoulai.com/presentations/...understand_Convolutional_Neural_Netw… · TLDR: Neural Networks are powerful but complex and

Aleksei [email protected]

+82 10 3742 3945