cmsa harvard a similarity-based ... - cmsa.fas.harvard.edu · information received by...
TRANSCRIPT
A similarity-based normative theory of biologically-plausible learning in neural networks
Cengiz ("Jen·ghiz") Pehlevan
Information received by
photoreceptors10 Gb/s
Retinal processing
10 Mb/s
Conscious awareness: 100 b/s
Raichle, Trends in Cog. Sci., 2010
World
“Information pyramid”Anderson et al., 2004
The brain processes data seamlessly and efficiently
Gradientdescent
Artificial
Wij<latexit sha1_base64="ldHPtZYaz6791fq+4J2SWN/H/9E=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RbDHghePFWwrtEvJptM2bTZZkqxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5QIbqzvf3uFjc2t7Z3ibmlv/+DwqHx80jIq1QybTAmlHyNqUHCJTcutwMdEI40jge1ocjuvt59QG67kg50mGMZ0KPmAM2qd1Wr3Mj6e9coVv+ovRNYhyKECuRq98le3r1gao7RMUGM6gZ/YMKPaciZwVuqmBhPKJnSIHYeSxmjCbLHtjFw4p08GSrsnLVm4vycyGhszjSPXGVM7Mqu1uflfrZPaQS3MuExSi5ItPxqkglhF5qeTPtfIrJg6oExztythI6opsy6gkgshWD15HVpX1cDx/XWlXsvjKMIZnMMlBHADdbiDBjSBwRie4RXePOW9eO/ex7K14OUzp/BH3ucPt32PLQ==</latexit><latexit sha1_base64="ldHPtZYaz6791fq+4J2SWN/H/9E=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RbDHghePFWwrtEvJptM2bTZZkqxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5QIbqzvf3uFjc2t7Z3ibmlv/+DwqHx80jIq1QybTAmlHyNqUHCJTcutwMdEI40jge1ocjuvt59QG67kg50mGMZ0KPmAM2qd1Wr3Mj6e9coVv+ovRNYhyKECuRq98le3r1gao7RMUGM6gZ/YMKPaciZwVuqmBhPKJnSIHYeSxmjCbLHtjFw4p08GSrsnLVm4vycyGhszjSPXGVM7Mqu1uflfrZPaQS3MuExSi5ItPxqkglhF5qeTPtfIrJg6oExztythI6opsy6gkgshWD15HVpX1cDx/XWlXsvjKMIZnMMlBHADdbiDBjSBwRie4RXePOW9eO/ex7K14OUzp/BH3ucPt32PLQ==</latexit><latexit sha1_base64="ldHPtZYaz6791fq+4J2SWN/H/9E=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RbDHghePFWwrtEvJptM2bTZZkqxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5QIbqzvf3uFjc2t7Z3ibmlv/+DwqHx80jIq1QybTAmlHyNqUHCJTcutwMdEI40jge1ocjuvt59QG67kg50mGMZ0KPmAM2qd1Wr3Mj6e9coVv+ovRNYhyKECuRq98le3r1gao7RMUGM6gZ/YMKPaciZwVuqmBhPKJnSIHYeSxmjCbLHtjFw4p08GSrsnLVm4vycyGhszjSPXGVM7Mqu1uflfrZPaQS3MuExSi5ItPxqkglhF5qeTPtfIrJg6oExztythI6opsy6gkgshWD15HVpX1cDx/XWlXsvjKMIZnMMlBHADdbiDBjSBwRie4RXePOW9eO/ex7K14OUzp/BH3ucPt32PLQ==</latexit><latexit sha1_base64="ldHPtZYaz6791fq+4J2SWN/H/9E=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RbDHghePFWwrtEvJptM2bTZZkqxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5QIbqzvf3uFjc2t7Z3ibmlv/+DwqHx80jIq1QybTAmlHyNqUHCJTcutwMdEI40jge1ocjuvt59QG67kg50mGMZ0KPmAM2qd1Wr3Mj6e9coVv+ovRNYhyKECuRq98le3r1gao7RMUGM6gZ/YMKPaciZwVuqmBhPKJnSIHYeSxmjCbLHtjFw4p08GSrsnLVm4vycyGhszjSPXGVM7Mqu1uflfrZPaQS3MuExSi5ItPxqkglhF5qeTPtfIrJg6oExztythI6opsy6gkgshWD15HVpX1cDx/XWlXsvjKMIZnMMlBHADdbiDBjSBwRie4RXePOW9eO/ex7K14OUzp/BH3ucPt32PLQ==</latexit>
Biology
Local!
�Wij = g (yi, xj ,Wij)<latexit sha1_base64="rto32cCEzMhdVw5+OIX+4RSbWNQ=">AAACFXicbZDJSgNBEIZ74h63qEcvjUGIEGRGBD0oCHrwqGBMIDMMPZ2apE3PQneNOAx5CS++ihcPingVvPk2dpaD2w8NH39VUV1/kEqh0bY/rdLU9Mzs3PxCeXFpeWW1srZ+rZNMcWjwRCaqFTANUsTQQIESWqkCFgUSmkH/dFhv3oLSIomvME/Bi1g3FqHgDI3lV+ruGUhktOkX4mZAj2nXlRBiLfdF/c6/qY99V4luD3f8StXetUeif8GZQJVMdOFXPtxOwrMIYuSSad127BS9gikUXMKg7GYaUsb7rAttgzGLQHvF6KoB3TZOh4aJMi9GOnK/TxQs0jqPAtMZMezp37Wh+V+tnWF46BUiTjOEmI8XhZmkmNBhRLQjFHCUuQHGlTB/pbzHFONogiybEJzfJ/+F671dx/DlfvXkaBLHPNkkW6RGHHJATsg5uSANwsk9eSTP5MV6sJ6sV+tt3FqyJjMb5Ies9y9pS55V</latexit><latexit sha1_base64="rto32cCEzMhdVw5+OIX+4RSbWNQ=">AAACFXicbZDJSgNBEIZ74h63qEcvjUGIEGRGBD0oCHrwqGBMIDMMPZ2apE3PQneNOAx5CS++ihcPingVvPk2dpaD2w8NH39VUV1/kEqh0bY/rdLU9Mzs3PxCeXFpeWW1srZ+rZNMcWjwRCaqFTANUsTQQIESWqkCFgUSmkH/dFhv3oLSIomvME/Bi1g3FqHgDI3lV+ruGUhktOkX4mZAj2nXlRBiLfdF/c6/qY99V4luD3f8StXetUeif8GZQJVMdOFXPtxOwrMIYuSSad127BS9gikUXMKg7GYaUsb7rAttgzGLQHvF6KoB3TZOh4aJMi9GOnK/TxQs0jqPAtMZMezp37Wh+V+tnWF46BUiTjOEmI8XhZmkmNBhRLQjFHCUuQHGlTB/pbzHFONogiybEJzfJ/+F671dx/DlfvXkaBLHPNkkW6RGHHJATsg5uSANwsk9eSTP5MV6sJ6sV+tt3FqyJjMb5Ies9y9pS55V</latexit><latexit sha1_base64="rto32cCEzMhdVw5+OIX+4RSbWNQ=">AAACFXicbZDJSgNBEIZ74h63qEcvjUGIEGRGBD0oCHrwqGBMIDMMPZ2apE3PQneNOAx5CS++ihcPingVvPk2dpaD2w8NH39VUV1/kEqh0bY/rdLU9Mzs3PxCeXFpeWW1srZ+rZNMcWjwRCaqFTANUsTQQIESWqkCFgUSmkH/dFhv3oLSIomvME/Bi1g3FqHgDI3lV+ruGUhktOkX4mZAj2nXlRBiLfdF/c6/qY99V4luD3f8StXetUeif8GZQJVMdOFXPtxOwrMIYuSSad127BS9gikUXMKg7GYaUsb7rAttgzGLQHvF6KoB3TZOh4aJMi9GOnK/TxQs0jqPAtMZMezp37Wh+V+tnWF46BUiTjOEmI8XhZmkmNBhRLQjFHCUuQHGlTB/pbzHFONogiybEJzfJ/+F671dx/DlfvXkaBLHPNkkW6RGHHJATsg5uSANwsk9eSTP5MV6sJ6sV+tt3FqyJjMb5Ies9y9pS55V</latexit><latexit sha1_base64="rto32cCEzMhdVw5+OIX+4RSbWNQ=">AAACFXicbZDJSgNBEIZ74h63qEcvjUGIEGRGBD0oCHrwqGBMIDMMPZ2apE3PQneNOAx5CS++ihcPingVvPk2dpaD2w8NH39VUV1/kEqh0bY/rdLU9Mzs3PxCeXFpeWW1srZ+rZNMcWjwRCaqFTANUsTQQIESWqkCFgUSmkH/dFhv3oLSIomvME/Bi1g3FqHgDI3lV+ruGUhktOkX4mZAj2nXlRBiLfdF/c6/qY99V4luD3f8StXetUeif8GZQJVMdOFXPtxOwrMIYuSSad127BS9gikUXMKg7GYaUsb7rAttgzGLQHvF6KoB3TZOh4aJMi9GOnK/TxQs0jqPAtMZMezp37Wh+V+tnWF46BUiTjOEmI8XhZmkmNBhRLQjFHCUuQHGlTB/pbzHFONogiybEJzfJ/+F671dx/DlfvXkaBLHPNkkW6RGHHJATsg5uSANwsk9eSTP5MV6sJ6sV+tt3FqyJjMb5Ies9y9pS55V</latexit>
xj<latexit sha1_base64="CO1uldGagW/3IuyI16j3rdK367g=">AAAB7HicbZBNS8NAEIYnftb6VfXoZbEInkoigh48FLx4rGDaQhvKZrtp1242YXciltDf4MWDIl79Qd78N27bHLT1hYWHd2bYmTdMpTDout/Oyura+sZmaau8vbO7t185OGyaJNOM+yyRiW6H1HApFPdRoOTtVHMah5K3wtHNtN565NqIRN3jOOVBTAdKRIJRtJafP016D71K1a25M5Fl8AqoQqFGr/LV7Scsi7lCJqkxHc9NMcipRsEkn5S7meEpZSM64B2LisbcBPls2Qk5tU6fRIm2TyGZub8nchobM45D2xlTHJrF2tT8r9bJMLoKcqHSDLli84+iTBJMyPRy0heaM5RjC5RpYXclbEg1ZWjzKdsQvMWTl6F5XvMs311U69dFHCU4hhM4Aw8uoQ630AAfGAh4hld4c5Tz4rw7H/PWFaeYOYI/cj5/ACV2jt8=</latexit><latexit sha1_base64="CO1uldGagW/3IuyI16j3rdK367g=">AAAB7HicbZBNS8NAEIYnftb6VfXoZbEInkoigh48FLx4rGDaQhvKZrtp1242YXciltDf4MWDIl79Qd78N27bHLT1hYWHd2bYmTdMpTDout/Oyura+sZmaau8vbO7t185OGyaJNOM+yyRiW6H1HApFPdRoOTtVHMah5K3wtHNtN565NqIRN3jOOVBTAdKRIJRtJafP016D71K1a25M5Fl8AqoQqFGr/LV7Scsi7lCJqkxHc9NMcipRsEkn5S7meEpZSM64B2LisbcBPls2Qk5tU6fRIm2TyGZub8nchobM45D2xlTHJrF2tT8r9bJMLoKcqHSDLli84+iTBJMyPRy0heaM5RjC5RpYXclbEg1ZWjzKdsQvMWTl6F5XvMs311U69dFHCU4hhM4Aw8uoQ630AAfGAh4hld4c5Tz4rw7H/PWFaeYOYI/cj5/ACV2jt8=</latexit><latexit sha1_base64="CO1uldGagW/3IuyI16j3rdK367g=">AAAB7HicbZBNS8NAEIYnftb6VfXoZbEInkoigh48FLx4rGDaQhvKZrtp1242YXciltDf4MWDIl79Qd78N27bHLT1hYWHd2bYmTdMpTDout/Oyura+sZmaau8vbO7t185OGyaJNOM+yyRiW6H1HApFPdRoOTtVHMah5K3wtHNtN565NqIRN3jOOVBTAdKRIJRtJafP016D71K1a25M5Fl8AqoQqFGr/LV7Scsi7lCJqkxHc9NMcipRsEkn5S7meEpZSM64B2LisbcBPls2Qk5tU6fRIm2TyGZub8nchobM45D2xlTHJrF2tT8r9bJMLoKcqHSDLli84+iTBJMyPRy0heaM5RjC5RpYXclbEg1ZWjzKdsQvMWTl6F5XvMs311U69dFHCU4hhM4Aw8uoQ630AAfGAh4hld4c5Tz4rw7H/PWFaeYOYI/cj5/ACV2jt8=</latexit><latexit sha1_base64="CO1uldGagW/3IuyI16j3rdK367g=">AAAB7HicbZBNS8NAEIYnftb6VfXoZbEInkoigh48FLx4rGDaQhvKZrtp1242YXciltDf4MWDIl79Qd78N27bHLT1hYWHd2bYmTdMpTDout/Oyura+sZmaau8vbO7t185OGyaJNOM+yyRiW6H1HApFPdRoOTtVHMah5K3wtHNtN565NqIRN3jOOVBTAdKRIJRtJafP016D71K1a25M5Fl8AqoQqFGr/LV7Scsi7lCJqkxHc9NMcipRsEkn5S7meEpZSM64B2LisbcBPls2Qk5tU6fRIm2TyGZub8nchobM45D2xlTHJrF2tT8r9bJMLoKcqHSDLli84+iTBJMyPRy0heaM5RjC5RpYXclbEg1ZWjzKdsQvMWTl6F5XvMs311U69dFHCU4hhM4Aw8uoQ630AAfGAh4hld4c5Tz4rw7H/PWFaeYOYI/cj5/ACV2jt8=</latexit>
yi<latexit sha1_base64="hnfuRit7EsxI7s8mTVQ+wehKqvY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfTgoeDFYwVTC20om+2mXbrZhN2JEEJ/gxcPinj1B3nz37htc9DWFxYe3plhZ94wlcKg6347lbX1jc2t6nZtZ3dv/6B+eNQxSaYZ91kiE90NqeFSKO6jQMm7qeY0DiV/DCe3s/rjE9dGJOoB85QHMR0pEQlG0Vp+kU8HYlBvuE13LrIKXgkNKNUe1L/6w4RlMVfIJDWm57kpBgXVKJjk01o/MzylbEJHvGdR0ZiboJgvOyVn1hmSKNH2KSRz9/dEQWNj8ji0nTHFsVmuzcz/ar0Mo+ugECrNkCu2+CjKJMGEzC4nQ6E5Q5lboEwLuythY6opQ5tPzYbgLZ+8Cp2Lpmf5/rLRuinjqMIJnMI5eHAFLbiDNvjAQMAzvMKbo5wX5935WLRWnHLmGP7I+fwBJXmO3w==</latexit><latexit sha1_base64="hnfuRit7EsxI7s8mTVQ+wehKqvY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfTgoeDFYwVTC20om+2mXbrZhN2JEEJ/gxcPinj1B3nz37htc9DWFxYe3plhZ94wlcKg6347lbX1jc2t6nZtZ3dv/6B+eNQxSaYZ91kiE90NqeFSKO6jQMm7qeY0DiV/DCe3s/rjE9dGJOoB85QHMR0pEQlG0Vp+kU8HYlBvuE13LrIKXgkNKNUe1L/6w4RlMVfIJDWm57kpBgXVKJjk01o/MzylbEJHvGdR0ZiboJgvOyVn1hmSKNH2KSRz9/dEQWNj8ji0nTHFsVmuzcz/ar0Mo+ugECrNkCu2+CjKJMGEzC4nQ6E5Q5lboEwLuythY6opQ5tPzYbgLZ+8Cp2Lpmf5/rLRuinjqMIJnMI5eHAFLbiDNvjAQMAzvMKbo5wX5935WLRWnHLmGP7I+fwBJXmO3w==</latexit><latexit sha1_base64="hnfuRit7EsxI7s8mTVQ+wehKqvY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfTgoeDFYwVTC20om+2mXbrZhN2JEEJ/gxcPinj1B3nz37htc9DWFxYe3plhZ94wlcKg6347lbX1jc2t6nZtZ3dv/6B+eNQxSaYZ91kiE90NqeFSKO6jQMm7qeY0DiV/DCe3s/rjE9dGJOoB85QHMR0pEQlG0Vp+kU8HYlBvuE13LrIKXgkNKNUe1L/6w4RlMVfIJDWm57kpBgXVKJjk01o/MzylbEJHvGdR0ZiboJgvOyVn1hmSKNH2KSRz9/dEQWNj8ji0nTHFsVmuzcz/ar0Mo+ugECrNkCu2+CjKJMGEzC4nQ6E5Q5lboEwLuythY6opQ5tPzYbgLZ+8Cp2Lpmf5/rLRuinjqMIJnMI5eHAFLbiDNvjAQMAzvMKbo5wX5935WLRWnHLmGP7I+fwBJXmO3w==</latexit><latexit sha1_base64="hnfuRit7EsxI7s8mTVQ+wehKqvY=">AAAB7HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfTgoeDFYwVTC20om+2mXbrZhN2JEEJ/gxcPinj1B3nz37htc9DWFxYe3plhZ94wlcKg6347lbX1jc2t6nZtZ3dv/6B+eNQxSaYZ91kiE90NqeFSKO6jQMm7qeY0DiV/DCe3s/rjE9dGJOoB85QHMR0pEQlG0Vp+kU8HYlBvuE13LrIKXgkNKNUe1L/6w4RlMVfIJDWm57kpBgXVKJjk01o/MzylbEJHvGdR0ZiboJgvOyVn1hmSKNH2KSRz9/dEQWNj8ji0nTHFsVmuzcz/ar0Mo+ugECrNkCu2+CjKJMGEzC4nQ6E5Q5lboEwLuythY6opQ5tPzYbgLZ+8Cp2Lpmf5/rLRuinjqMIJnMI5eHAFLbiDNvjAQMAzvMKbo5wX5935WLRWnHLmGP7I+fwBJXmO3w==</latexit>
Non-local!
�Wij =@C
@Wij<latexit sha1_base64="584oLPGWt54tlI9g3Is19YXQTWw=">AAACHHicbZDLSsNAFIYnXmu9RV26GSyCq5KooAuFQl24rGAv0IQymZ60YycXZiZCCXkQN76KGxeKuHEh+DZO2oDaemDg4//PmZnzezFnUlnWl7GwuLS8slpaK69vbG5tmzu7LRklgkKTRjwSHY9I4CyEpmKKQycWQAKPQ9sb1XO/fQ9Csii8VeMY3IAMQuYzSpSWeuaJcwVcEdzupewuw5fY8QWhqRMToRjhuJ798LQnwz2zYlWtSeF5sAuooKIaPfPD6Uc0CSBUlBMpu7YVKzfNr6UcsrKTSIgJHZEBdDWGJADpppPlMnyolT72I6FPqPBE/T2RkkDKceDpzoCooZz1cvE/r5so/9xNWRgnCkI6fchPOFYRzpPCfSaAKj7WQKhg+q+YDolOR+k8yzoEe3bleWgdV23NN6eV2kURRwntowN0hGx0hmroGjVQE1H0gJ7QC3o1Ho1n4814n7YuGMXMHvpTxuc3jLKhpQ==</latexit><latexit sha1_base64="584oLPGWt54tlI9g3Is19YXQTWw=">AAACHHicbZDLSsNAFIYnXmu9RV26GSyCq5KooAuFQl24rGAv0IQymZ60YycXZiZCCXkQN76KGxeKuHEh+DZO2oDaemDg4//PmZnzezFnUlnWl7GwuLS8slpaK69vbG5tmzu7LRklgkKTRjwSHY9I4CyEpmKKQycWQAKPQ9sb1XO/fQ9Csii8VeMY3IAMQuYzSpSWeuaJcwVcEdzupewuw5fY8QWhqRMToRjhuJ798LQnwz2zYlWtSeF5sAuooKIaPfPD6Uc0CSBUlBMpu7YVKzfNr6UcsrKTSIgJHZEBdDWGJADpppPlMnyolT72I6FPqPBE/T2RkkDKceDpzoCooZz1cvE/r5so/9xNWRgnCkI6fchPOFYRzpPCfSaAKj7WQKhg+q+YDolOR+k8yzoEe3bleWgdV23NN6eV2kURRwntowN0hGx0hmroGjVQE1H0gJ7QC3o1Ho1n4814n7YuGMXMHvpTxuc3jLKhpQ==</latexit><latexit sha1_base64="584oLPGWt54tlI9g3Is19YXQTWw=">AAACHHicbZDLSsNAFIYnXmu9RV26GSyCq5KooAuFQl24rGAv0IQymZ60YycXZiZCCXkQN76KGxeKuHEh+DZO2oDaemDg4//PmZnzezFnUlnWl7GwuLS8slpaK69vbG5tmzu7LRklgkKTRjwSHY9I4CyEpmKKQycWQAKPQ9sb1XO/fQ9Csii8VeMY3IAMQuYzSpSWeuaJcwVcEdzupewuw5fY8QWhqRMToRjhuJ798LQnwz2zYlWtSeF5sAuooKIaPfPD6Uc0CSBUlBMpu7YVKzfNr6UcsrKTSIgJHZEBdDWGJADpppPlMnyolT72I6FPqPBE/T2RkkDKceDpzoCooZz1cvE/r5so/9xNWRgnCkI6fchPOFYRzpPCfSaAKj7WQKhg+q+YDolOR+k8yzoEe3bleWgdV23NN6eV2kURRwntowN0hGx0hmroGjVQE1H0gJ7QC3o1Ho1n4814n7YuGMXMHvpTxuc3jLKhpQ==</latexit><latexit sha1_base64="584oLPGWt54tlI9g3Is19YXQTWw=">AAACHHicbZDLSsNAFIYnXmu9RV26GSyCq5KooAuFQl24rGAv0IQymZ60YycXZiZCCXkQN76KGxeKuHEh+DZO2oDaemDg4//PmZnzezFnUlnWl7GwuLS8slpaK69vbG5tmzu7LRklgkKTRjwSHY9I4CyEpmKKQycWQAKPQ9sb1XO/fQ9Csii8VeMY3IAMQuYzSpSWeuaJcwVcEdzupewuw5fY8QWhqRMToRjhuJ798LQnwz2zYlWtSeF5sAuooKIaPfPD6Uc0CSBUlBMpu7YVKzfNr6UcsrKTSIgJHZEBdDWGJADpppPlMnyolT72I6FPqPBE/T2RkkDKceDpzoCooZz1cvE/r5so/9xNWRgnCkI6fchPOFYRzpPCfSaAKj7WQKhg+q+YDolOR+k8yzoEe3bleWgdV23NN6eV2kURRwntowN0hGx0hmroGjVQE1H0gJ7QC3o1Ho1n4814n7YuGMXMHvpTxuc3jLKhpQ==</latexit>
C<latexit sha1_base64="BmccXKpURRUGK6dATqi/3LhQU1I=">AAAB6HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbSwCKSxTMB8QHKEvc1csmZv79jdE8KRX2BjoYitP8nOf+MmuUITX1h4eGeGnXmDRHBtXPfbKWxsbm3vFHdLe/sHh0fl45O2jlPFsMViEatuQDUKLrFluBHYTRTSKBDYCSb1eb3zhErzWD6YaYJ+REeSh5xRY61mfVCuuFV3IbIOXg4VyNUYlL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LMoaYTazxaLzsiFdYYkjJV90pCF+3sio5HW0yiwnRE1Y71am5v/1XqpCW/9jMskNSjZ8qMwFcTEZH41GXKFzIipBcoUt7sSNqaKMmOzKdkQvNWT16F9VfUsN68rtbs8jiKcwTlcggc3UIN7aEALGCA8wyu8OY/Oi/PufCxbC04+cwp/5Hz+AJQZjME=</latexit><latexit sha1_base64="BmccXKpURRUGK6dATqi/3LhQU1I=">AAAB6HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbSwCKSxTMB8QHKEvc1csmZv79jdE8KRX2BjoYitP8nOf+MmuUITX1h4eGeGnXmDRHBtXPfbKWxsbm3vFHdLe/sHh0fl45O2jlPFsMViEatuQDUKLrFluBHYTRTSKBDYCSb1eb3zhErzWD6YaYJ+REeSh5xRY61mfVCuuFV3IbIOXg4VyNUYlL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LMoaYTazxaLzsiFdYYkjJV90pCF+3sio5HW0yiwnRE1Y71am5v/1XqpCW/9jMskNSjZ8qMwFcTEZH41GXKFzIipBcoUt7sSNqaKMmOzKdkQvNWT16F9VfUsN68rtbs8jiKcwTlcggc3UIN7aEALGCA8wyu8OY/Oi/PufCxbC04+cwp/5Hz+AJQZjME=</latexit><latexit sha1_base64="BmccXKpURRUGK6dATqi/3LhQU1I=">AAAB6HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbSwCKSxTMB8QHKEvc1csmZv79jdE8KRX2BjoYitP8nOf+MmuUITX1h4eGeGnXmDRHBtXPfbKWxsbm3vFHdLe/sHh0fl45O2jlPFsMViEatuQDUKLrFluBHYTRTSKBDYCSb1eb3zhErzWD6YaYJ+REeSh5xRY61mfVCuuFV3IbIOXg4VyNUYlL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LMoaYTazxaLzsiFdYYkjJV90pCF+3sio5HW0yiwnRE1Y71am5v/1XqpCW/9jMskNSjZ8qMwFcTEZH41GXKFzIipBcoUt7sSNqaKMmOzKdkQvNWT16F9VfUsN68rtbs8jiKcwTlcggc3UIN7aEALGCA8wyu8OY/Oi/PufCxbC04+cwp/5Hz+AJQZjME=</latexit><latexit sha1_base64="BmccXKpURRUGK6dATqi/3LhQU1I=">AAAB6HicbZA9SwNBEIbn4leMX1FLm8UgWIU7EbSwCKSxTMB8QHKEvc1csmZv79jdE8KRX2BjoYitP8nOf+MmuUITX1h4eGeGnXmDRHBtXPfbKWxsbm3vFHdLe/sHh0fl45O2jlPFsMViEatuQDUKLrFluBHYTRTSKBDYCSb1eb3zhErzWD6YaYJ+REeSh5xRY61mfVCuuFV3IbIOXg4VyNUYlL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LMoaYTazxaLzsiFdYYkjJV90pCF+3sio5HW0yiwnRE1Y71am5v/1XqpCW/9jMskNSjZ8qMwFcTEZH41GXKFzIipBcoUt7sSNqaKMmOzKdkQvNWT16F9VfUsN68rtbs8jiKcwTlcggc3UIN7aEALGCA8wyu8OY/Oi/PufCxbC04+cwp/5Hz+AJQZjME=</latexit>Cost function
Are we missing something?
1. Cost function
2. Gradient descent
What can we learn from how the brain trains its networks?
NNs with local learning rules
Architecture, dynamics,learning rules…
Local neural networks (NNs) from similarity-based cost functions
Gradientlearning
Biologically motivatedsimilarity-based cost
functions
“Representation is representation of similarities” Shimon Edelman, 1998
Images Representation
2
66664
512. . .. . .. . .
3
77775
2
66664
202. . .. . .. . .
3
77775
Similarity Similarity
“Representation is representation of similarities” Shimon Edelman, 1998
Kriegeskorte et. al., 2008
Human IT CortexMonkey IT Cortex
Similarity
Similarity-based cost function for PCA: Similarity-matching
Input: Output:x1,x2, . . . ,xT<latexit sha1_base64="5gUc1zvhuI1tBPdV0WlNaOhdoBY=">AAACDXicbZDLSgMxFIYzXmu9jbp0E6yCi1JmiqALFwU3Liv0Bp0yZDKZNjSTDElGLENfwI2v4saFIm7du/NtTNtRtPWHwMd/zuHk/EHCqNKO82ktLa+srq0XNoqbW9s7u/befkuJVGLSxIIJ2QmQIoxy0tRUM9JJJEFxwEg7GF5N6u1bIhUVvKFHCenFqM9pRDHSxvLt48wLIng39t3yN1XLHguFVj9Gw7dLTsWZCi6Cm0MJ5Kr79ocXCpzGhGvMkFJd10l0L0NSU8zIuOiliiQID1GfdA1yFBPVy6bXjOGJcUIYCWke13Dq/p7IUKzUKA5MZ4z0QM3XJuZ/tW6qo4teRnmSasLxbFGUMqgFnEQDQyoJ1mxkAGFJzV8hHiCJsDYBFk0I7vzJi9CqVlzDN2el2mUeRwEcgiNwClxwDmrgGtRBE2BwDx7BM3ixHqwn69V6m7UuWfnMAfgj6/0LQEWbAw==</latexit><latexit sha1_base64="5gUc1zvhuI1tBPdV0WlNaOhdoBY=">AAACDXicbZDLSgMxFIYzXmu9jbp0E6yCi1JmiqALFwU3Liv0Bp0yZDKZNjSTDElGLENfwI2v4saFIm7du/NtTNtRtPWHwMd/zuHk/EHCqNKO82ktLa+srq0XNoqbW9s7u/befkuJVGLSxIIJ2QmQIoxy0tRUM9JJJEFxwEg7GF5N6u1bIhUVvKFHCenFqM9pRDHSxvLt48wLIng39t3yN1XLHguFVj9Gw7dLTsWZCi6Cm0MJ5Kr79ocXCpzGhGvMkFJd10l0L0NSU8zIuOiliiQID1GfdA1yFBPVy6bXjOGJcUIYCWke13Dq/p7IUKzUKA5MZ4z0QM3XJuZ/tW6qo4teRnmSasLxbFGUMqgFnEQDQyoJ1mxkAGFJzV8hHiCJsDYBFk0I7vzJi9CqVlzDN2el2mUeRwEcgiNwClxwDmrgGtRBE2BwDx7BM3ixHqwn69V6m7UuWfnMAfgj6/0LQEWbAw==</latexit><latexit sha1_base64="5gUc1zvhuI1tBPdV0WlNaOhdoBY=">AAACDXicbZDLSgMxFIYzXmu9jbp0E6yCi1JmiqALFwU3Liv0Bp0yZDKZNjSTDElGLENfwI2v4saFIm7du/NtTNtRtPWHwMd/zuHk/EHCqNKO82ktLa+srq0XNoqbW9s7u/befkuJVGLSxIIJ2QmQIoxy0tRUM9JJJEFxwEg7GF5N6u1bIhUVvKFHCenFqM9pRDHSxvLt48wLIng39t3yN1XLHguFVj9Gw7dLTsWZCi6Cm0MJ5Kr79ocXCpzGhGvMkFJd10l0L0NSU8zIuOiliiQID1GfdA1yFBPVy6bXjOGJcUIYCWke13Dq/p7IUKzUKA5MZ4z0QM3XJuZ/tW6qo4teRnmSasLxbFGUMqgFnEQDQyoJ1mxkAGFJzV8hHiCJsDYBFk0I7vzJi9CqVlzDN2el2mUeRwEcgiNwClxwDmrgGtRBE2BwDx7BM3ixHqwn69V6m7UuWfnMAfgj6/0LQEWbAw==</latexit><latexit sha1_base64="5gUc1zvhuI1tBPdV0WlNaOhdoBY=">AAACDXicbZDLSgMxFIYzXmu9jbp0E6yCi1JmiqALFwU3Liv0Bp0yZDKZNjSTDElGLENfwI2v4saFIm7du/NtTNtRtPWHwMd/zuHk/EHCqNKO82ktLa+srq0XNoqbW9s7u/befkuJVGLSxIIJ2QmQIoxy0tRUM9JJJEFxwEg7GF5N6u1bIhUVvKFHCenFqM9pRDHSxvLt48wLIng39t3yN1XLHguFVj9Gw7dLTsWZCi6Cm0MJ5Kr79ocXCpzGhGvMkFJd10l0L0NSU8zIuOiliiQID1GfdA1yFBPVy6bXjOGJcUIYCWke13Dq/p7IUKzUKA5MZ4z0QM3XJuZ/tW6qo4teRnmSasLxbFGUMqgFnEQDQyoJ1mxkAGFJzV8hHiCJsDYBFk0I7vzJi9CqVlzDN2el2mUeRwEcgiNwClxwDmrgGtRBE2BwDx7BM3ixHqwn69V6m7UuWfnMAfgj6/0LQEWbAw==</latexit>
y1,y2, . . . ,yT<latexit sha1_base64="MbXJV9/eOCM9+my7f/57QfeAFog=">AAACDXicbZDLSsNAFIYn9VbrLerSzWAVXJSSFEEXLgpuXFboDZoQJpNJO3SSCTMTIYS+gBtfxY0LRdy6d+fbOG2jaOsPAx//OYcz5/cTRqWyrE+jtLK6tr5R3qxsbe/s7pn7B13JU4FJB3PGRd9HkjAak46iipF+IgiKfEZ6/vh6Wu/dESEpj9sqS4gboWFMQ4qR0pZnnuSOH8Js4tm1b2rUHBZwJX+MtmdWrbo1E1wGu4AqKNTyzA8n4DiNSKwwQ1IObCtRbo6EopiRScVJJUkQHqMhGWiMUUSkm8+umcBT7QQw5EK/WMGZ+3siR5GUWeTrzgipkVysTc3/aoNUhZduTuMkVSTG80VhyqDicBoNDKggWLFMA8KC6r9CPEICYaUDrOgQ7MWTl6HbqNuab8+rzasijjI4AsfgDNjgAjTBDWiBDsDgHjyCZ/BiPBhPxqvxNm8tGcXMIfgj4/0LRQabBg==</latexit><latexit sha1_base64="MbXJV9/eOCM9+my7f/57QfeAFog=">AAACDXicbZDLSsNAFIYn9VbrLerSzWAVXJSSFEEXLgpuXFboDZoQJpNJO3SSCTMTIYS+gBtfxY0LRdy6d+fbOG2jaOsPAx//OYcz5/cTRqWyrE+jtLK6tr5R3qxsbe/s7pn7B13JU4FJB3PGRd9HkjAak46iipF+IgiKfEZ6/vh6Wu/dESEpj9sqS4gboWFMQ4qR0pZnnuSOH8Js4tm1b2rUHBZwJX+MtmdWrbo1E1wGu4AqKNTyzA8n4DiNSKwwQ1IObCtRbo6EopiRScVJJUkQHqMhGWiMUUSkm8+umcBT7QQw5EK/WMGZ+3siR5GUWeTrzgipkVysTc3/aoNUhZduTuMkVSTG80VhyqDicBoNDKggWLFMA8KC6r9CPEICYaUDrOgQ7MWTl6HbqNuab8+rzasijjI4AsfgDNjgAjTBDWiBDsDgHjyCZ/BiPBhPxqvxNm8tGcXMIfgj4/0LRQabBg==</latexit><latexit sha1_base64="MbXJV9/eOCM9+my7f/57QfeAFog=">AAACDXicbZDLSsNAFIYn9VbrLerSzWAVXJSSFEEXLgpuXFboDZoQJpNJO3SSCTMTIYS+gBtfxY0LRdy6d+fbOG2jaOsPAx//OYcz5/cTRqWyrE+jtLK6tr5R3qxsbe/s7pn7B13JU4FJB3PGRd9HkjAak46iipF+IgiKfEZ6/vh6Wu/dESEpj9sqS4gboWFMQ4qR0pZnnuSOH8Js4tm1b2rUHBZwJX+MtmdWrbo1E1wGu4AqKNTyzA8n4DiNSKwwQ1IObCtRbo6EopiRScVJJUkQHqMhGWiMUUSkm8+umcBT7QQw5EK/WMGZ+3siR5GUWeTrzgipkVysTc3/aoNUhZduTuMkVSTG80VhyqDicBoNDKggWLFMA8KC6r9CPEICYaUDrOgQ7MWTl6HbqNuab8+rzasijjI4AsfgDNjgAjTBDWiBDsDgHjyCZ/BiPBhPxqvxNm8tGcXMIfgj4/0LRQabBg==</latexit><latexit sha1_base64="MbXJV9/eOCM9+my7f/57QfeAFog=">AAACDXicbZDLSsNAFIYn9VbrLerSzWAVXJSSFEEXLgpuXFboDZoQJpNJO3SSCTMTIYS+gBtfxY0LRdy6d+fbOG2jaOsPAx//OYcz5/cTRqWyrE+jtLK6tr5R3qxsbe/s7pn7B13JU4FJB3PGRd9HkjAak46iipF+IgiKfEZ6/vh6Wu/dESEpj9sqS4gboWFMQ4qR0pZnnuSOH8Js4tm1b2rUHBZwJX+MtmdWrbo1E1wGu4AqKNTyzA8n4DiNSKwwQ1IObCtRbo6EopiRScVJJUkQHqMhGWiMUUSkm8+umcBT7QQw5EK/WMGZ+3siR5GUWeTrzgipkVysTc3/aoNUhZduTuMkVSTG80VhyqDicBoNDKggWLFMA8KC6r9CPEICYaUDrOgQ7MWTl6HbqNuab8+rzasijjI4AsfgDNjgAjTBDWiBDsDgHjyCZ/BiPBhPxqvxNm8tGcXMIfgj4/0LRQabBg==</latexit>
Input similarity
Output similarity
Similarity Matching (Classical Multidimensional Scaling):
miny1,...,yT
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0
�2
<latexit sha1_base64="e2CYm1r4zu+jGT9hPwI3yQj21Hw=">AAACbHicbVFdSxtBFJ1da9WoNVUfWqQwGNQIKrtSsA8WhL70USFRIZsss5PZZHA+lpm7pcuyT/5D3/oTfPE3OEk24EcvDJx7zrnzcSbJBLcQBP88f+HD4sel5ZXG6tr6p43m581rq3NDWZdqoc1tQiwTXLEucBDsNjOMyESwm+Tu10S/+cOM5Vp1oMhYX5KR4imnBBwVN+8jyVVcllGS4iIOq6NIDDXYoxlRxZ0KRzaXcQk/w2rQmTcHdSdYCu2p928VwyACnc07Z6rwMZ5v9EIsZmJk+GgMh4PTuNkKToJp4fcgrEEL1XUZNx+ioaa5ZAqoINb2wiCDfkkMcCpY1YhyyzJC78iI9RxURDLbL6dhVXjPMUOcauOWAjxlX06URFpbyMQ5JYGxfatNyP9pvRzSH/2SqywHpujsoDQXGDSeJI+H3DAKonCAUMPdXTEdE0MouP9puBDCt09+D65PT0KHr763Ls7rOJbRDtpFbRSiM3SBfqNL1EUUPXob3hfvq/fkb/s7/reZ1ffqmS30qvz9Z6W1u0w=</latexit><latexit sha1_base64="e2CYm1r4zu+jGT9hPwI3yQj21Hw=">AAACbHicbVFdSxtBFJ1da9WoNVUfWqQwGNQIKrtSsA8WhL70USFRIZsss5PZZHA+lpm7pcuyT/5D3/oTfPE3OEk24EcvDJx7zrnzcSbJBLcQBP88f+HD4sel5ZXG6tr6p43m581rq3NDWZdqoc1tQiwTXLEucBDsNjOMyESwm+Tu10S/+cOM5Vp1oMhYX5KR4imnBBwVN+8jyVVcllGS4iIOq6NIDDXYoxlRxZ0KRzaXcQk/w2rQmTcHdSdYCu2p928VwyACnc07Z6rwMZ5v9EIsZmJk+GgMh4PTuNkKToJp4fcgrEEL1XUZNx+ioaa5ZAqoINb2wiCDfkkMcCpY1YhyyzJC78iI9RxURDLbL6dhVXjPMUOcauOWAjxlX06URFpbyMQ5JYGxfatNyP9pvRzSH/2SqywHpujsoDQXGDSeJI+H3DAKonCAUMPdXTEdE0MouP9puBDCt09+D65PT0KHr763Ls7rOJbRDtpFbRSiM3SBfqNL1EUUPXob3hfvq/fkb/s7/reZ1ffqmS30qvz9Z6W1u0w=</latexit><latexit sha1_base64="e2CYm1r4zu+jGT9hPwI3yQj21Hw=">AAACbHicbVFdSxtBFJ1da9WoNVUfWqQwGNQIKrtSsA8WhL70USFRIZsss5PZZHA+lpm7pcuyT/5D3/oTfPE3OEk24EcvDJx7zrnzcSbJBLcQBP88f+HD4sel5ZXG6tr6p43m581rq3NDWZdqoc1tQiwTXLEucBDsNjOMyESwm+Tu10S/+cOM5Vp1oMhYX5KR4imnBBwVN+8jyVVcllGS4iIOq6NIDDXYoxlRxZ0KRzaXcQk/w2rQmTcHdSdYCu2p928VwyACnc07Z6rwMZ5v9EIsZmJk+GgMh4PTuNkKToJp4fcgrEEL1XUZNx+ioaa5ZAqoINb2wiCDfkkMcCpY1YhyyzJC78iI9RxURDLbL6dhVXjPMUOcauOWAjxlX06URFpbyMQ5JYGxfatNyP9pvRzSH/2SqywHpujsoDQXGDSeJI+H3DAKonCAUMPdXTEdE0MouP9puBDCt09+D65PT0KHr763Ls7rOJbRDtpFbRSiM3SBfqNL1EUUPXob3hfvq/fkb/s7/reZ1ffqmS30qvz9Z6W1u0w=</latexit><latexit sha1_base64="e2CYm1r4zu+jGT9hPwI3yQj21Hw=">AAACbHicbVFdSxtBFJ1da9WoNVUfWqQwGNQIKrtSsA8WhL70USFRIZsss5PZZHA+lpm7pcuyT/5D3/oTfPE3OEk24EcvDJx7zrnzcSbJBLcQBP88f+HD4sel5ZXG6tr6p43m581rq3NDWZdqoc1tQiwTXLEucBDsNjOMyESwm+Tu10S/+cOM5Vp1oMhYX5KR4imnBBwVN+8jyVVcllGS4iIOq6NIDDXYoxlRxZ0KRzaXcQk/w2rQmTcHdSdYCu2p928VwyACnc07Z6rwMZ5v9EIsZmJk+GgMh4PTuNkKToJp4fcgrEEL1XUZNx+ioaa5ZAqoINb2wiCDfkkMcCpY1YhyyzJC78iI9RxURDLbL6dhVXjPMUOcauOWAjxlX06URFpbyMQ5JYGxfatNyP9pvRzSH/2SqywHpujsoDQXGDSeJI+H3DAKonCAUMPdXTEdE0MouP9puBDCt09+D65PT0KHr763Ls7rOJbRDtpFbRSiM3SBfqNL1EUUPXob3hfvq/fkb/s7/reZ1ffqmS30qvz9Z6W1u0w=</latexit>
x1<latexit sha1_base64="Hh8eKWh0+bzhJBHCvfeo5e3IdGw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSESqeCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qe/1yxW36s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn81Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xwys+ESlLkii0+ClNJMCazu8lAaM5QTixQpoXdlbAR1ZShTadkQ/CWT16F1kXVs3x3Walf53EU4QRO4Rw8qEEdbqEBTWAwhGd4hTdHOi/Ou/OxaC04+cwx/JHz+QMJSo2Y</latexit><latexit sha1_base64="Hh8eKWh0+bzhJBHCvfeo5e3IdGw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSESqeCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qe/1yxW36s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn81Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xwys+ESlLkii0+ClNJMCazu8lAaM5QTixQpoXdlbAR1ZShTadkQ/CWT16F1kXVs3x3Walf53EU4QRO4Rw8qEEdbqEBTWAwhGd4hTdHOi/Ou/OxaC04+cwx/JHz+QMJSo2Y</latexit><latexit sha1_base64="Hh8eKWh0+bzhJBHCvfeo5e3IdGw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSESqeCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qe/1yxW36s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn81Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xwys+ESlLkii0+ClNJMCazu8lAaM5QTixQpoXdlbAR1ZShTadkQ/CWT16F1kXVs3x3Walf53EU4QRO4Rw8qEEdbqEBTWAwhGd4hTdHOi/Ou/OxaC04+cwx/JHz+QMJSo2Y</latexit><latexit sha1_base64="Hh8eKWh0+bzhJBHCvfeo5e3IdGw=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSESqeCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qe/1yxW36s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn81Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xwys+ESlLkii0+ClNJMCazu8lAaM5QTixQpoXdlbAR1ZShTadkQ/CWT16F1kXVs3x3Walf53EU4QRO4Rw8qEEdbqEBTWAwhGd4hTdHOi/Ou/OxaC04+cwx/JHz+QMJSo2Y</latexit>
x2<latexit sha1_base64="/9POIrDH85nMhhpXpXxMnPLSEM4=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIiieCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qV/rlytu1Z2LrIKXQwVyNfrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWlQ04sbP5qtOyZl1BiSMtX0Kydz9PZHRyJhJFNjOiOLILNdm5n+1borhlZ8JlaTIFVt8FKaSYExmd5OB0JyhnFigTAu7K2EjqilDm07JhuAtn7wKrVrVs3x3Ualf53EU4QRO4Rw8uIQ63EIDmsBgCM/wCm+OdF6cd+dj0Vpw8plj+CPn8wcKzo2Z</latexit><latexit sha1_base64="/9POIrDH85nMhhpXpXxMnPLSEM4=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIiieCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qV/rlytu1Z2LrIKXQwVyNfrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWlQ04sbP5qtOyZl1BiSMtX0Kydz9PZHRyJhJFNjOiOLILNdm5n+1borhlZ8JlaTIFVt8FKaSYExmd5OB0JyhnFigTAu7K2EjqilDm07JhuAtn7wKrVrVs3x3Ualf53EU4QRO4Rw8uIQ63EIDmsBgCM/wCm+OdF6cd+dj0Vpw8plj+CPn8wcKzo2Z</latexit><latexit sha1_base64="/9POIrDH85nMhhpXpXxMnPLSEM4=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIiieCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qV/rlytu1Z2LrIKXQwVyNfrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWlQ04sbP5qtOyZl1BiSMtX0Kydz9PZHRyJhJFNjOiOLILNdm5n+1borhlZ8JlaTIFVt8FKaSYExmd5OB0JyhnFigTAu7K2EjqilDm07JhuAtn7wKrVrVs3x3Ualf53EU4QRO4Rw8uIQ63EIDmsBgCM/wCm+OdF6cd+dj0Vpw8plj+CPn8wcKzo2Z</latexit><latexit sha1_base64="/9POIrDH85nMhhpXpXxMnPLSEM4=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FSSIiieCl48VrQf0Iay2W7apZtN2J2IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cm1ErB5wknA/okMlQsEoWuv+qV/rlytu1Z2LrIKXQwVyNfrlr94gZmnEFTJJjel6boJ+RjUKJvm01EsNTygb0yHvWlQ04sbP5qtOyZl1BiSMtX0Kydz9PZHRyJhJFNjOiOLILNdm5n+1borhlZ8JlaTIFVt8FKaSYExmd5OB0JyhnFigTAu7K2EjqilDm07JhuAtn7wKrVrVs3x3Ualf53EU4QRO4Rw8uIQ63EIDmsBgCM/wCm+OdF6cd+dj0Vpw8plj+CPn8wcKzo2Z</latexit>
x3<latexit sha1_base64="VxhWG6QECTNIjgyWUA9tN7S39ZQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSFRRPBS8eK9oPaEPZbDft0s0m7E7EEvoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJFAZd99sprKyurW8UN0tb2zu7e+X9g6aJU814g8Uy1u2AGi6F4g0UKHk70ZxGgeStYHQzrbceuTYiVg84Trgf0YESoWAUrXX/1DvvlStu1Z2JLIOXQwVy1Xvlr24/ZmnEFTJJjel4boJ+RjUKJvmk1E0NTygb0QHvWFQ04sbPZqtOyIl1+iSMtX0Kycz9PZHRyJhxFNjOiOLQLNam5n+1TorhlZ8JlaTIFZt/FKaSYEymd5O+0JyhHFugTAu7K2FDqilDm07JhuAtnrwMzbOqZ/nuolK7zuMowhEcwyl4cAk1uIU6NIDBAJ7hFd4c6bw4787HvLXg5DOH8EfO5w8MUo2a</latexit><latexit sha1_base64="VxhWG6QECTNIjgyWUA9tN7S39ZQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSFRRPBS8eK9oPaEPZbDft0s0m7E7EEvoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJFAZd99sprKyurW8UN0tb2zu7e+X9g6aJU814g8Uy1u2AGi6F4g0UKHk70ZxGgeStYHQzrbceuTYiVg84Trgf0YESoWAUrXX/1DvvlStu1Z2JLIOXQwVy1Xvlr24/ZmnEFTJJjel4boJ+RjUKJvmk1E0NTygb0QHvWFQ04sbPZqtOyIl1+iSMtX0Kycz9PZHRyJhxFNjOiOLQLNam5n+1TorhlZ8JlaTIFZt/FKaSYEymd5O+0JyhHFugTAu7K2FDqilDm07JhuAtnrwMzbOqZ/nuolK7zuMowhEcwyl4cAk1uIU6NIDBAJ7hFd4c6bw4787HvLXg5DOH8EfO5w8MUo2a</latexit><latexit sha1_base64="VxhWG6QECTNIjgyWUA9tN7S39ZQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSFRRPBS8eK9oPaEPZbDft0s0m7E7EEvoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJFAZd99sprKyurW8UN0tb2zu7e+X9g6aJU814g8Uy1u2AGi6F4g0UKHk70ZxGgeStYHQzrbceuTYiVg84Trgf0YESoWAUrXX/1DvvlStu1Z2JLIOXQwVy1Xvlr24/ZmnEFTJJjel4boJ+RjUKJvmk1E0NTygb0QHvWFQ04sbPZqtOyIl1+iSMtX0Kycz9PZHRyJhxFNjOiOLQLNam5n+1TorhlZ8JlaTIFZt/FKaSYEymd5O+0JyhHFugTAu7K2FDqilDm07JhuAtnrwMzbOqZ/nuolK7zuMowhEcwyl4cAk1uIU6NIDBAJ7hFd4c6bw4787HvLXg5DOH8EfO5w8MUo2a</latexit><latexit sha1_base64="VxhWG6QECTNIjgyWUA9tN7S39ZQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSFRRPBS8eK9oPaEPZbDft0s0m7E7EEvoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJFAZd99sprKyurW8UN0tb2zu7e+X9g6aJU814g8Uy1u2AGi6F4g0UKHk70ZxGgeStYHQzrbceuTYiVg84Trgf0YESoWAUrXX/1DvvlStu1Z2JLIOXQwVy1Xvlr24/ZmnEFTJJjel4boJ+RjUKJvmk1E0NTygb0QHvWFQ04sbPZqtOyIl1+iSMtX0Kycz9PZHRyJhxFNjOiOLQLNam5n+1TorhlZ8JlaTIFZt/FKaSYEymd5O+0JyhHFugTAu7K2FDqilDm07JhuAtnrwMzbOqZ/nuolK7zuMowhEcwyl4cAk1uIU6NIDBAJ7hFd4c6bw4787HvLXg5DOH8EfO5w8MUo2a</latexit>
y1<latexit sha1_base64="+2FDDgr3tS764PjeV8VwZk0mBEQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSERRPBS8eK9oPaEPZbCft0s0m7G6EEPoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJ4Nq47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4phi8UiVt2AahRcYstwI7CbKKRRILATTG5n9c4TKs1j+WiyBP2IjiQPOaPGWg/ZwBtUa27dnYusgldADQo1B9Wv/jBmaYTSMEG17nluYvycKsOZwGmln2pMKJvQEfYsShqh9vP5qlNyZp0hCWNlnzRk7v6eyGmkdRYFtjOiZqyXazPzv1ovNeG1n3OZpAYlW3wUpoKYmMzuJkOukBmRWaBMcbsrYWOqKDM2nYoNwVs+eRXaF3XP8v1lrXFTxFGGEziFc/DgChpwB01oAYMRPMMrvDnCeXHenY9Fa8kpZo7hj5zPHwrQjZk=</latexit><latexit sha1_base64="+2FDDgr3tS764PjeV8VwZk0mBEQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSERRPBS8eK9oPaEPZbCft0s0m7G6EEPoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJ4Nq47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4phi8UiVt2AahRcYstwI7CbKKRRILATTG5n9c4TKs1j+WiyBP2IjiQPOaPGWg/ZwBtUa27dnYusgldADQo1B9Wv/jBmaYTSMEG17nluYvycKsOZwGmln2pMKJvQEfYsShqh9vP5qlNyZp0hCWNlnzRk7v6eyGmkdRYFtjOiZqyXazPzv1ovNeG1n3OZpAYlW3wUpoKYmMzuJkOukBmRWaBMcbsrYWOqKDM2nYoNwVs+eRXaF3XP8v1lrXFTxFGGEziFc/DgChpwB01oAYMRPMMrvDnCeXHenY9Fa8kpZo7hj5zPHwrQjZk=</latexit><latexit sha1_base64="+2FDDgr3tS764PjeV8VwZk0mBEQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSERRPBS8eK9oPaEPZbCft0s0m7G6EEPoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJ4Nq47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4phi8UiVt2AahRcYstwI7CbKKRRILATTG5n9c4TKs1j+WiyBP2IjiQPOaPGWg/ZwBtUa27dnYusgldADQo1B9Wv/jBmaYTSMEG17nluYvycKsOZwGmln2pMKJvQEfYsShqh9vP5qlNyZp0hCWNlnzRk7v6eyGmkdRYFtjOiZqyXazPzv1ovNeG1n3OZpAYlW3wUpoKYmMzuJkOukBmRWaBMcbsrYWOqKDM2nYoNwVs+eRXaF3XP8v1lrXFTxFGGEziFc/DgChpwB01oAYMRPMMrvDnCeXHenY9Fa8kpZo7hj5zPHwrQjZk=</latexit><latexit sha1_base64="+2FDDgr3tS764PjeV8VwZk0mBEQ=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSERRPBS8eK9oPaEPZbCft0s0m7G6EEPoTvHhQxKu/yJv/xm2bg7a+sPDwzgw78waJ4Nq47rdTWlvf2Nwqb1d2dvf2D6qHR20dp4phi8UiVt2AahRcYstwI7CbKKRRILATTG5n9c4TKs1j+WiyBP2IjiQPOaPGWg/ZwBtUa27dnYusgldADQo1B9Wv/jBmaYTSMEG17nluYvycKsOZwGmln2pMKJvQEfYsShqh9vP5qlNyZp0hCWNlnzRk7v6eyGmkdRYFtjOiZqyXazPzv1ovNeG1n3OZpAYlW3wUpoKYmMzuJkOukBmRWaBMcbsrYWOqKDM2nYoNwVs+eRXaF3XP8v1lrXFTxFGGEziFc/DgChpwB01oAYMRPMMrvDnCeXHenY9Fa8kpZo7hj5zPHwrQjZk=</latexit>
y2<latexit sha1_base64="M/i0KoeJET7WNKR/5tw3FfIuTlo=">AAAB6nicbZBNS8NAEIYnftb6VfXoZbEInkpSBMVTwYvHivYD2lA22027dLMJuxMhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XbW1jc2t7ZLO+Xdvf2Dw8rRcdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmd1TtPXBsRq0fMEu5HdKREKBhFaz1kg/qgUnVr7lxkFbwCqlCoOah89YcxSyOukElqTM9zE/RzqlEwyaflfmp4QtmEjnjPoqIRN34+X3VKzq0zJGGs7VNI5u7viZxGxmRRYDsjimOzXJuZ/9V6KYbXfi5UkiJXbPFRmEqCMZndTYZCc4Yys0CZFnZXwsZUU4Y2nbINwVs+eRXa9Zpn+f6y2rgp4ijBKZzBBXhwBQ24gya0gMEInuEV3hzpvDjvzseidc0pZk7gj5zPHwxUjZo=</latexit><latexit sha1_base64="M/i0KoeJET7WNKR/5tw3FfIuTlo=">AAAB6nicbZBNS8NAEIYnftb6VfXoZbEInkpSBMVTwYvHivYD2lA22027dLMJuxMhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XbW1jc2t7ZLO+Xdvf2Dw8rRcdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmd1TtPXBsRq0fMEu5HdKREKBhFaz1kg/qgUnVr7lxkFbwCqlCoOah89YcxSyOukElqTM9zE/RzqlEwyaflfmp4QtmEjnjPoqIRN34+X3VKzq0zJGGs7VNI5u7viZxGxmRRYDsjimOzXJuZ/9V6KYbXfi5UkiJXbPFRmEqCMZndTYZCc4Yys0CZFnZXwsZUU4Y2nbINwVs+eRXa9Zpn+f6y2rgp4ijBKZzBBXhwBQ24gya0gMEInuEV3hzpvDjvzseidc0pZk7gj5zPHwxUjZo=</latexit><latexit sha1_base64="M/i0KoeJET7WNKR/5tw3FfIuTlo=">AAAB6nicbZBNS8NAEIYnftb6VfXoZbEInkpSBMVTwYvHivYD2lA22027dLMJuxMhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XbW1jc2t7ZLO+Xdvf2Dw8rRcdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmd1TtPXBsRq0fMEu5HdKREKBhFaz1kg/qgUnVr7lxkFbwCqlCoOah89YcxSyOukElqTM9zE/RzqlEwyaflfmp4QtmEjnjPoqIRN34+X3VKzq0zJGGs7VNI5u7viZxGxmRRYDsjimOzXJuZ/9V6KYbXfi5UkiJXbPFRmEqCMZndTYZCc4Yys0CZFnZXwsZUU4Y2nbINwVs+eRXa9Zpn+f6y2rgp4ijBKZzBBXhwBQ24gya0gMEInuEV3hzpvDjvzseidc0pZk7gj5zPHwxUjZo=</latexit><latexit sha1_base64="M/i0KoeJET7WNKR/5tw3FfIuTlo=">AAAB6nicbZBNS8NAEIYnftb6VfXoZbEInkpSBMVTwYvHivYD2lA22027dLMJuxMhhP4ELx4U8eov8ua/cdvmoK0vLDy8M8POvEEihUHX/XbW1jc2t7ZLO+Xdvf2Dw8rRcdvEqWa8xWIZ625ADZdC8RYKlLybaE6jQPJOMLmd1TtPXBsRq0fMEu5HdKREKBhFaz1kg/qgUnVr7lxkFbwCqlCoOah89YcxSyOukElqTM9zE/RzqlEwyaflfmp4QtmEjnjPoqIRN34+X3VKzq0zJGGs7VNI5u7viZxGxmRRYDsjimOzXJuZ/9V6KYbXfi5UkiJXbPFRmEqCMZndTYZCc4Yys0CZFnZXwsZUU4Y2nbINwVs+eRXa9Zpn+f6y2rgp4ijBKZzBBXhwBQ24gya0gMEInuEV3hzpvDjvzseidc0pZk7gj5zPHwxUjZo=</latexit>
2 Rn<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
2 Rk<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
Steps towards a local NN
Pehlevan, Hu & Chklovskii., Neural Comp., 2015Pehlevan, Sengupta & Chklovskii, Neural Comp., 2018
Similarity Matching miny1,...,yT
1
T 2
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0
�2
<latexit sha1_base64="nZTKZ3teXbzSMp6RFzSrONo4b9w=">AAACeHicbVFNaxsxENVu2zRx28RtjrmIuCEJpGHXFFoKhUAvPSZgJwHLXrSy1hbWxyLNli6LfkP/W2/9IbnkFNneQD46IHjz3sxIepOXUjhIkn9R/OLlq43Xm1udN2/fbe9033+4dKayjA+ZkcZe59RxKTQfggDJr0vLqcolv8oXP5b61S9unTB6AHXJx4rOtCgEoxCorPuHKKGzpiF5gess9SdETg24kzXhs4HHpLCU4bQZTPqeuEplDXxP/WSA2+SwzSQv4GjV99tnMCFgyvssFHn8Cd8PfSDWa5FYMZvD8aSfdXvJabIK/BykLeihNs6z7l8yNaxSXAOT1LlRmpQwbqgFwST3HVI5XlK2oDM+ClBTxd24WRnn8UFgprgwNhwNeMU+7Giocq5WeahUFObuqbYk/6eNKii+jhuhywq4ZuuLikpiMHi5BTwVljOQdQCUWRHeitmcBp8h7KoTTEiffvk5uOyfpgFffO6dfWvt2ER7aB8doRR9QWfoJzpHQ8TQTbQXfYwOotsYx4fx8bo0jtqeXfQo4v4d90S/5w==</latexit><latexit sha1_base64="nZTKZ3teXbzSMp6RFzSrONo4b9w=">AAACeHicbVFNaxsxENVu2zRx28RtjrmIuCEJpGHXFFoKhUAvPSZgJwHLXrSy1hbWxyLNli6LfkP/W2/9IbnkFNneQD46IHjz3sxIepOXUjhIkn9R/OLlq43Xm1udN2/fbe9033+4dKayjA+ZkcZe59RxKTQfggDJr0vLqcolv8oXP5b61S9unTB6AHXJx4rOtCgEoxCorPuHKKGzpiF5gess9SdETg24kzXhs4HHpLCU4bQZTPqeuEplDXxP/WSA2+SwzSQv4GjV99tnMCFgyvssFHn8Cd8PfSDWa5FYMZvD8aSfdXvJabIK/BykLeihNs6z7l8yNaxSXAOT1LlRmpQwbqgFwST3HVI5XlK2oDM+ClBTxd24WRnn8UFgprgwNhwNeMU+7Giocq5WeahUFObuqbYk/6eNKii+jhuhywq4ZuuLikpiMHi5BTwVljOQdQCUWRHeitmcBp8h7KoTTEiffvk5uOyfpgFffO6dfWvt2ER7aB8doRR9QWfoJzpHQ8TQTbQXfYwOotsYx4fx8bo0jtqeXfQo4v4d90S/5w==</latexit><latexit sha1_base64="nZTKZ3teXbzSMp6RFzSrONo4b9w=">AAACeHicbVFNaxsxENVu2zRx28RtjrmIuCEJpGHXFFoKhUAvPSZgJwHLXrSy1hbWxyLNli6LfkP/W2/9IbnkFNneQD46IHjz3sxIepOXUjhIkn9R/OLlq43Xm1udN2/fbe9033+4dKayjA+ZkcZe59RxKTQfggDJr0vLqcolv8oXP5b61S9unTB6AHXJx4rOtCgEoxCorPuHKKGzpiF5gess9SdETg24kzXhs4HHpLCU4bQZTPqeuEplDXxP/WSA2+SwzSQv4GjV99tnMCFgyvssFHn8Cd8PfSDWa5FYMZvD8aSfdXvJabIK/BykLeihNs6z7l8yNaxSXAOT1LlRmpQwbqgFwST3HVI5XlK2oDM+ClBTxd24WRnn8UFgprgwNhwNeMU+7Giocq5WeahUFObuqbYk/6eNKii+jhuhywq4ZuuLikpiMHi5BTwVljOQdQCUWRHeitmcBp8h7KoTTEiffvk5uOyfpgFffO6dfWvt2ER7aB8doRR9QWfoJzpHQ8TQTbQXfYwOotsYx4fx8bo0jtqeXfQo4v4d90S/5w==</latexit><latexit sha1_base64="nZTKZ3teXbzSMp6RFzSrONo4b9w=">AAACeHicbVFNaxsxENVu2zRx28RtjrmIuCEJpGHXFFoKhUAvPSZgJwHLXrSy1hbWxyLNli6LfkP/W2/9IbnkFNneQD46IHjz3sxIepOXUjhIkn9R/OLlq43Xm1udN2/fbe9033+4dKayjA+ZkcZe59RxKTQfggDJr0vLqcolv8oXP5b61S9unTB6AHXJx4rOtCgEoxCorPuHKKGzpiF5gess9SdETg24kzXhs4HHpLCU4bQZTPqeuEplDXxP/WSA2+SwzSQv4GjV99tnMCFgyvssFHn8Cd8PfSDWa5FYMZvD8aSfdXvJabIK/BykLeihNs6z7l8yNaxSXAOT1LlRmpQwbqgFwST3HVI5XlK2oDM+ClBTxd24WRnn8UFgprgwNhwNeMU+7Giocq5WeahUFObuqbYk/6eNKii+jhuhywq4ZuuLikpiMHi5BTwVljOQdQCUWRHeitmcBp8h7KoTTEiffvk5uOyfpgFffO6dfWvt2ER7aB8doRR9QWfoJzpHQ8TQTbQXfYwOotsYx4fx8bo0jtqeXfQo4v4d90S/5w==</latexit>
miny1,...,yT
1
T 2
TX
t=1
"�2x>
t
TX
t0=1
xt0y>t0
!yt + y
>t
TX
t0=1
yt0y>t0
!yt
#
<latexit sha1_base64="ogrEwk2vey/8vRgKMU1UO7ZkdJs=">AAAC1nicjZJdi9QwFIbT+rWOX7N66U1wEFdcl3YQFEFY8MbLFWd2ByadkmbSmbBpUpLTxVLihSLe+tu880f4H8y0HdFdQQ8U3jznPSftOc1KKSxE0fcgvHT5ytVrO9cHN27eun1nuHv32OrKMD5lWmozy6jlUig+BQGSz0rDaZFJfpKdvt7kT864sUKrCdQlTwq6UiIXjIJH6fAHKYRKm4ZkOa7T2O0TudRg9zvg0onDJDeU4biZLMaO2KpIG3gVu8UEE8lzmD8dt973LoUFAV12eK93PuqsW4sHbtt6o9sKYsRqDY+3HPAT/Ev/o2X9ny27c5IOR9FB1Aa+KOJejFAfR+nwG1lqVhVcAZPU2nkclZA01IBgkrsBqSwvKTulKz73UtGC26Rp1+LwQ0+WONfGPwpwS3+vaGhhbV1k3llQWNvzuQ38W25eQf4iaYQqK+CKdRfllcSg8WbHeCkMZyBrLygzwr8rZmvqtwj+Txj4IcTnP/miOB4fxF6/fTY6fNmPYwfdRw/QHorRc3SI3qAjNEUseBfUwcfgUzgLP4Sfwy+dNQz6mnvojwi//gQrCeUI</latexit><latexit sha1_base64="ogrEwk2vey/8vRgKMU1UO7ZkdJs=">AAAC1nicjZJdi9QwFIbT+rWOX7N66U1wEFdcl3YQFEFY8MbLFWd2ByadkmbSmbBpUpLTxVLihSLe+tu880f4H8y0HdFdQQ8U3jznPSftOc1KKSxE0fcgvHT5ytVrO9cHN27eun1nuHv32OrKMD5lWmozy6jlUig+BQGSz0rDaZFJfpKdvt7kT864sUKrCdQlTwq6UiIXjIJH6fAHKYRKm4ZkOa7T2O0TudRg9zvg0onDJDeU4biZLMaO2KpIG3gVu8UEE8lzmD8dt973LoUFAV12eK93PuqsW4sHbtt6o9sKYsRqDY+3HPAT/Ev/o2X9ny27c5IOR9FB1Aa+KOJejFAfR+nwG1lqVhVcAZPU2nkclZA01IBgkrsBqSwvKTulKz73UtGC26Rp1+LwQ0+WONfGPwpwS3+vaGhhbV1k3llQWNvzuQ38W25eQf4iaYQqK+CKdRfllcSg8WbHeCkMZyBrLygzwr8rZmvqtwj+Txj4IcTnP/miOB4fxF6/fTY6fNmPYwfdRw/QHorRc3SI3qAjNEUseBfUwcfgUzgLP4Sfwy+dNQz6mnvojwi//gQrCeUI</latexit><latexit sha1_base64="ogrEwk2vey/8vRgKMU1UO7ZkdJs=">AAAC1nicjZJdi9QwFIbT+rWOX7N66U1wEFdcl3YQFEFY8MbLFWd2ByadkmbSmbBpUpLTxVLihSLe+tu880f4H8y0HdFdQQ8U3jznPSftOc1KKSxE0fcgvHT5ytVrO9cHN27eun1nuHv32OrKMD5lWmozy6jlUig+BQGSz0rDaZFJfpKdvt7kT864sUKrCdQlTwq6UiIXjIJH6fAHKYRKm4ZkOa7T2O0TudRg9zvg0onDJDeU4biZLMaO2KpIG3gVu8UEE8lzmD8dt973LoUFAV12eK93PuqsW4sHbtt6o9sKYsRqDY+3HPAT/Ev/o2X9ny27c5IOR9FB1Aa+KOJejFAfR+nwG1lqVhVcAZPU2nkclZA01IBgkrsBqSwvKTulKz73UtGC26Rp1+LwQ0+WONfGPwpwS3+vaGhhbV1k3llQWNvzuQ38W25eQf4iaYQqK+CKdRfllcSg8WbHeCkMZyBrLygzwr8rZmvqtwj+Txj4IcTnP/miOB4fxF6/fTY6fNmPYwfdRw/QHorRc3SI3qAjNEUseBfUwcfgUzgLP4Sfwy+dNQz6mnvojwi//gQrCeUI</latexit><latexit sha1_base64="ogrEwk2vey/8vRgKMU1UO7ZkdJs=">AAAC1nicjZJdi9QwFIbT+rWOX7N66U1wEFdcl3YQFEFY8MbLFWd2ByadkmbSmbBpUpLTxVLihSLe+tu880f4H8y0HdFdQQ8U3jznPSftOc1KKSxE0fcgvHT5ytVrO9cHN27eun1nuHv32OrKMD5lWmozy6jlUig+BQGSz0rDaZFJfpKdvt7kT864sUKrCdQlTwq6UiIXjIJH6fAHKYRKm4ZkOa7T2O0TudRg9zvg0onDJDeU4biZLMaO2KpIG3gVu8UEE8lzmD8dt973LoUFAV12eK93PuqsW4sHbtt6o9sKYsRqDY+3HPAT/Ev/o2X9ny27c5IOR9FB1Aa+KOJejFAfR+nwG1lqVhVcAZPU2nkclZA01IBgkrsBqSwvKTulKz73UtGC26Rp1+LwQ0+WONfGPwpwS3+vaGhhbV1k3llQWNvzuQ38W25eQf4iaYQqK+CKdRfllcSg8WbHeCkMZyBrLygzwr8rZmvqtwj+Txj4IcTnP/miOB4fxF6/fTY6fNmPYwfdRw/QHorRc3SI3qAjNEUseBfUwcfgUzgLP4Sfwy+dNQz6mnvojwi//gQrCeUI</latexit>
minW
� 4
T
TX
t=1
x
>t W
>yt + 2TrW>
W
<latexit sha1_base64="/X7VNevtlWekwGfefDN6cTtL5sA=">AAACV3icbVHRSiMxFM2MXbd23bWuj75cLMKCrMyIoCwIgi8+KrS20GmHTJrRYJIZkjvLlmF+UnzxV/ZF0+k8aOuFwLnnnMtNTpJcCotB8OL5G60vm1/bW51v299/7HR3f97ZrDCMD1gmMzNKqOVSaD5AgZKPcsOpSiQfJo9XC334lxsrMt3Hec4nit5rkQpG0VFxV0dK6LiMkhSGFfyOUkMZlKdVHyCyhYpLvAiraR9qx78qxmmEWd74a7yU5k6CIzhxnVHQNxWse9yCuNsLjoO6YB2EDeiRpm7i7lM0y1ihuEYmqbXjMMhxUlKDgkledaLC8pyyR3rPxw5qqridlHUuFRw6ZgZpZtzRCDX7fqKkytq5SpxTUXywq9qC/EwbF5ieT0qh8wK5ZstFaSEBM1iEDDNhOEM5d4AyI9xdgT1QFy26r+i4EMLVJ6+Du5Pj0OHb097lnyaONtknB+QXCckZuSTX5IYMCCPP5L+34bW8F+/V3/TbS6vvNTN75EP5u28BBLKs</latexit><latexit sha1_base64="/X7VNevtlWekwGfefDN6cTtL5sA=">AAACV3icbVHRSiMxFM2MXbd23bWuj75cLMKCrMyIoCwIgi8+KrS20GmHTJrRYJIZkjvLlmF+UnzxV/ZF0+k8aOuFwLnnnMtNTpJcCotB8OL5G60vm1/bW51v299/7HR3f97ZrDCMD1gmMzNKqOVSaD5AgZKPcsOpSiQfJo9XC334lxsrMt3Hec4nit5rkQpG0VFxV0dK6LiMkhSGFfyOUkMZlKdVHyCyhYpLvAiraR9qx78qxmmEWd74a7yU5k6CIzhxnVHQNxWse9yCuNsLjoO6YB2EDeiRpm7i7lM0y1ihuEYmqbXjMMhxUlKDgkledaLC8pyyR3rPxw5qqridlHUuFRw6ZgZpZtzRCDX7fqKkytq5SpxTUXywq9qC/EwbF5ieT0qh8wK5ZstFaSEBM1iEDDNhOEM5d4AyI9xdgT1QFy26r+i4EMLVJ6+Du5Pj0OHb097lnyaONtknB+QXCckZuSTX5IYMCCPP5L+34bW8F+/V3/TbS6vvNTN75EP5u28BBLKs</latexit><latexit sha1_base64="/X7VNevtlWekwGfefDN6cTtL5sA=">AAACV3icbVHRSiMxFM2MXbd23bWuj75cLMKCrMyIoCwIgi8+KrS20GmHTJrRYJIZkjvLlmF+UnzxV/ZF0+k8aOuFwLnnnMtNTpJcCotB8OL5G60vm1/bW51v299/7HR3f97ZrDCMD1gmMzNKqOVSaD5AgZKPcsOpSiQfJo9XC334lxsrMt3Hec4nit5rkQpG0VFxV0dK6LiMkhSGFfyOUkMZlKdVHyCyhYpLvAiraR9qx78qxmmEWd74a7yU5k6CIzhxnVHQNxWse9yCuNsLjoO6YB2EDeiRpm7i7lM0y1ihuEYmqbXjMMhxUlKDgkledaLC8pyyR3rPxw5qqridlHUuFRw6ZgZpZtzRCDX7fqKkytq5SpxTUXywq9qC/EwbF5ieT0qh8wK5ZstFaSEBM1iEDDNhOEM5d4AyI9xdgT1QFy26r+i4EMLVJ6+Du5Pj0OHb097lnyaONtknB+QXCckZuSTX5IYMCCPP5L+34bW8F+/V3/TbS6vvNTN75EP5u28BBLKs</latexit><latexit sha1_base64="/X7VNevtlWekwGfefDN6cTtL5sA=">AAACV3icbVHRSiMxFM2MXbd23bWuj75cLMKCrMyIoCwIgi8+KrS20GmHTJrRYJIZkjvLlmF+UnzxV/ZF0+k8aOuFwLnnnMtNTpJcCotB8OL5G60vm1/bW51v299/7HR3f97ZrDCMD1gmMzNKqOVSaD5AgZKPcsOpSiQfJo9XC334lxsrMt3Hec4nit5rkQpG0VFxV0dK6LiMkhSGFfyOUkMZlKdVHyCyhYpLvAiraR9qx78qxmmEWd74a7yU5k6CIzhxnVHQNxWse9yCuNsLjoO6YB2EDeiRpm7i7lM0y1ihuEYmqbXjMMhxUlKDgkledaLC8pyyR3rPxw5qqridlHUuFRw6ZgZpZtzRCDX7fqKkytq5SpxTUXywq9qC/EwbF5ieT0qh8wK5ZstFaSEBM1iEDDNhOEM5d4AyI9xdgT1QFy26r+i4EMLVJ6+Du5Pj0OHb097lnyaONtknB+QXCckZuSTX5IYMCCPP5L+34bW8F+/V3/TbS6vvNTN75EP5u28BBLKs</latexit>
max
L
2
T
TX
t=1
y>t Lyt � TrL>L
<latexit sha1_base64="r7R0sC1H2EA/9v+ZL8L4ACrmRXQ=">AAACTnicbVFLS8QwGEzX17q+Vj16+XARvCitCIogCF48eFDYVWG7ljSbajBpS/JVXEp/oRfx5s/w4kERzdYqvgYCk5n5SDIJUykMuu6DUxsZHRufqE82pqZnZuea8wsnJsk04x2WyESfhdRwKWLeQYGSn6WaUxVKfhpe7Q/902uujUjiNg5S3lP0IhaRYBStFDS5r+hNkPthBIcF+JGmDPKNog3gm0wFOe56xXkbysCgCPDcxySt4l8irNmAVtDWBVRemfvcQNBsuetuCfhLvIq0SIWjoHnv9xOWKR4jk9SYruem2MupRsEkLxp+ZnhK2RW94F1LY6q46eVlHQWsWKUPUaLtihFK9ftETpUxAxXapKJ4aX57Q/E/r5thtN3LRZxmyGP2cVCUScAEht1CX2jOUA4soUwLe1dgl9RWivYHGrYE7/eT/5KTjXXP8uPN1t5OVUedLJFlsko8skX2yAE5Ih3CyC15JM/kxblznpxX5+0jWnOqmUXyA7X6O7qFs3k=</latexit><latexit sha1_base64="r7R0sC1H2EA/9v+ZL8L4ACrmRXQ=">AAACTnicbVFLS8QwGEzX17q+Vj16+XARvCitCIogCF48eFDYVWG7ljSbajBpS/JVXEp/oRfx5s/w4kERzdYqvgYCk5n5SDIJUykMuu6DUxsZHRufqE82pqZnZuea8wsnJsk04x2WyESfhdRwKWLeQYGSn6WaUxVKfhpe7Q/902uujUjiNg5S3lP0IhaRYBStFDS5r+hNkPthBIcF+JGmDPKNog3gm0wFOe56xXkbysCgCPDcxySt4l8irNmAVtDWBVRemfvcQNBsuetuCfhLvIq0SIWjoHnv9xOWKR4jk9SYruem2MupRsEkLxp+ZnhK2RW94F1LY6q46eVlHQWsWKUPUaLtihFK9ftETpUxAxXapKJ4aX57Q/E/r5thtN3LRZxmyGP2cVCUScAEht1CX2jOUA4soUwLe1dgl9RWivYHGrYE7/eT/5KTjXXP8uPN1t5OVUedLJFlsko8skX2yAE5Ih3CyC15JM/kxblznpxX5+0jWnOqmUXyA7X6O7qFs3k=</latexit><latexit sha1_base64="r7R0sC1H2EA/9v+ZL8L4ACrmRXQ=">AAACTnicbVFLS8QwGEzX17q+Vj16+XARvCitCIogCF48eFDYVWG7ljSbajBpS/JVXEp/oRfx5s/w4kERzdYqvgYCk5n5SDIJUykMuu6DUxsZHRufqE82pqZnZuea8wsnJsk04x2WyESfhdRwKWLeQYGSn6WaUxVKfhpe7Q/902uujUjiNg5S3lP0IhaRYBStFDS5r+hNkPthBIcF+JGmDPKNog3gm0wFOe56xXkbysCgCPDcxySt4l8irNmAVtDWBVRemfvcQNBsuetuCfhLvIq0SIWjoHnv9xOWKR4jk9SYruem2MupRsEkLxp+ZnhK2RW94F1LY6q46eVlHQWsWKUPUaLtihFK9ftETpUxAxXapKJ4aX57Q/E/r5thtN3LRZxmyGP2cVCUScAEht1CX2jOUA4soUwLe1dgl9RWivYHGrYE7/eT/5KTjXXP8uPN1t5OVUedLJFlsko8skX2yAE5Ih3CyC15JM/kxblznpxX5+0jWnOqmUXyA7X6O7qFs3k=</latexit><latexit sha1_base64="r7R0sC1H2EA/9v+ZL8L4ACrmRXQ=">AAACTnicbVFLS8QwGEzX17q+Vj16+XARvCitCIogCF48eFDYVWG7ljSbajBpS/JVXEp/oRfx5s/w4kERzdYqvgYCk5n5SDIJUykMuu6DUxsZHRufqE82pqZnZuea8wsnJsk04x2WyESfhdRwKWLeQYGSn6WaUxVKfhpe7Q/902uujUjiNg5S3lP0IhaRYBStFDS5r+hNkPthBIcF+JGmDPKNog3gm0wFOe56xXkbysCgCPDcxySt4l8irNmAVtDWBVRemfvcQNBsuetuCfhLvIq0SIWjoHnv9xOWKR4jk9SYruem2MupRsEkLxp+ZnhK2RW94F1LY6q46eVlHQWsWKUPUaLtihFK9ftETpUxAxXapKJ4aX57Q/E/r5thtN3LRZxmyGP2cVCUScAEht1CX2jOUA4soUwLe1dgl9RWivYHGrYE7/eT/5KTjXXP8uPN1t5OVUedLJFlsko8skX2yAE5Ih3CyC15JM/kxblznpxX5+0jWnOqmUXyA7X6O7qFs3k=</latexit>
Oja, 1982; Foldiak, 1989; Pehlevan, Hu & Chklovskii, 2015; Pehlevan, Sengupta & Chklovskii, 2018; Seung and Zung, 2017
A Hebbian/Anti-Hebbian network
1)#Neural#activity:
x1
xn
. . .
y1
y2x2
yk
. . .
. . .�L
W
2)#Plasticity: Hebbian
An78Hebbian
Wij �Wij + ⌘ (yixj �Wij)
Lij � Lij + ⌘ (yiyj � Lij)
yt = Wxt � Lyt<latexit sha1_base64="LJYw/K/L/ewHJsW4iq2yLbnOwd4=">AAACHXicbZBLS8NAEMc3Pmt9RT16WSyCF0siBUUQCl48eKhgH9CGsNlu2qWbB7sTsYR8ES9+FS8eFPHgRfw2btMg2jqw7I//zDAzfy8WXIFlfRkLi0vLK6ultfL6xubWtrmz21JRIilr0khEsuMRxQQPWRM4CNaJJSOBJ1jbG11O8u07JhWPwlsYx8wJyCDkPqcEtOSatV4/Apz2PB+PMxfwxZTbWf7dT6TjqXSd/VS5ZsWqWnngebALqKAiGq75ocfQJGAhUEGU6tpWDE5KJHAqWFbuJYrFhI7IgHU1hiRgyknz6zJ8qJU+9iOpXwg4V393pCRQahx4ujIgMFSzuYn4X66bgH/mpDyME2AhnQ7yE4EhwhOrcJ9LRkGMNRAqud4V0yGRhII2tKxNsGdPnofWSdXWfFOr1M8LO0poHx2gI2SjU1RHV6iBmoiiB/SEXtCr8Wg8G2/G+7R0wSh69tCfMD6/Abj3oQo=</latexit><latexit sha1_base64="LJYw/K/L/ewHJsW4iq2yLbnOwd4=">AAACHXicbZBLS8NAEMc3Pmt9RT16WSyCF0siBUUQCl48eKhgH9CGsNlu2qWbB7sTsYR8ES9+FS8eFPHgRfw2btMg2jqw7I//zDAzfy8WXIFlfRkLi0vLK6ultfL6xubWtrmz21JRIilr0khEsuMRxQQPWRM4CNaJJSOBJ1jbG11O8u07JhWPwlsYx8wJyCDkPqcEtOSatV4/Apz2PB+PMxfwxZTbWf7dT6TjqXSd/VS5ZsWqWnngebALqKAiGq75ocfQJGAhUEGU6tpWDE5KJHAqWFbuJYrFhI7IgHU1hiRgyknz6zJ8qJU+9iOpXwg4V393pCRQahx4ujIgMFSzuYn4X66bgH/mpDyME2AhnQ7yE4EhwhOrcJ9LRkGMNRAqud4V0yGRhII2tKxNsGdPnofWSdXWfFOr1M8LO0poHx2gI2SjU1RHV6iBmoiiB/SEXtCr8Wg8G2/G+7R0wSh69tCfMD6/Abj3oQo=</latexit><latexit sha1_base64="LJYw/K/L/ewHJsW4iq2yLbnOwd4=">AAACHXicbZBLS8NAEMc3Pmt9RT16WSyCF0siBUUQCl48eKhgH9CGsNlu2qWbB7sTsYR8ES9+FS8eFPHgRfw2btMg2jqw7I//zDAzfy8WXIFlfRkLi0vLK6ultfL6xubWtrmz21JRIilr0khEsuMRxQQPWRM4CNaJJSOBJ1jbG11O8u07JhWPwlsYx8wJyCDkPqcEtOSatV4/Apz2PB+PMxfwxZTbWf7dT6TjqXSd/VS5ZsWqWnngebALqKAiGq75ocfQJGAhUEGU6tpWDE5KJHAqWFbuJYrFhI7IgHU1hiRgyknz6zJ8qJU+9iOpXwg4V393pCRQahx4ujIgMFSzuYn4X66bgH/mpDyME2AhnQ7yE4EhwhOrcJ9LRkGMNRAqud4V0yGRhII2tKxNsGdPnofWSdXWfFOr1M8LO0poHx2gI2SjU1RHV6iBmoiiB/SEXtCr8Wg8G2/G+7R0wSh69tCfMD6/Abj3oQo=</latexit><latexit sha1_base64="LJYw/K/L/ewHJsW4iq2yLbnOwd4=">AAACHXicbZBLS8NAEMc3Pmt9RT16WSyCF0siBUUQCl48eKhgH9CGsNlu2qWbB7sTsYR8ES9+FS8eFPHgRfw2btMg2jqw7I//zDAzfy8WXIFlfRkLi0vLK6ultfL6xubWtrmz21JRIilr0khEsuMRxQQPWRM4CNaJJSOBJ1jbG11O8u07JhWPwlsYx8wJyCDkPqcEtOSatV4/Apz2PB+PMxfwxZTbWf7dT6TjqXSd/VS5ZsWqWnngebALqKAiGq75ocfQJGAhUEGU6tpWDE5KJHAqWFbuJYrFhI7IgHU1hiRgyknz6zJ8qJU+9iOpXwg4V393pCRQahx4ujIgMFSzuYn4X66bgH/mpDyME2AhnQ7yE4EhwhOrcJ9LRkGMNRAqud4V0yGRhII2tKxNsGdPnofWSdXWfFOr1M8LO0poHx2gI2SjU1RHV6iBmoiiB/SEXtCr8Wg8G2/G+7R0wSh69tCfMD6/Abj3oQo=</latexit>
min
Wmax
L
1
T
TX
t=1
2TrW
>W � TrL
>L+min
yt
�4x
>t W
>yt + 2y
>t Lyt
�
<latexit sha1_base64="OtQJE5t17cn0q9fPDV2tBUgro+Q=">AAACwHicbZFda9swFIZl76vLPpptl7s5LAwGpcUOhY1BoWw3u8hFB0lTiFwjK3IiKtmedDyaGf3J3Yz9mymO12bpDgiec9736OtklZIWo+h3EN67/+Dho73HvSdPnz3f7794eW7L2nAx4aUqzUXGrFCyEBOUqMRFZQTTmRLT7OrzWp9+F8bKshjjqhKJZotC5pIz9KW0/4tqWaQNzXKYOqrZ9YZHDmhuGIcmdmOgttZpgyexuxxTJXKcAQyhoUbD2Djoui8pltXf5HBHHm3LI3cAm3PbdOVSdIfHLV97bp3bm9644GB4w7fKyN0agBq5WGKS9gfRUdQG3IW4gwHp4izt/6TzktdaFMgVs3YWRxUmDTMouRKuR2srKsav2ELMPBZMC5s07QAcvPWVOeSl8atAaKvbHQ3T1q505p2a4dLuauvi/7RZjfmHpJFFVaMo+OagvFaAJaynCXNpBEe18sC4kf6uwJfMDw79zHv+E+LdJ9+F8+FR7Pnr8eD0Y/cde+Q1eUPekZi8J6fkCzkjE8KDk4AHKtDhp3AZluG3jTUMup5X5J8If/wBdgTaAQ==</latexit><latexit sha1_base64="OtQJE5t17cn0q9fPDV2tBUgro+Q=">AAACwHicbZFda9swFIZl76vLPpptl7s5LAwGpcUOhY1BoWw3u8hFB0lTiFwjK3IiKtmedDyaGf3J3Yz9mymO12bpDgiec9736OtklZIWo+h3EN67/+Dho73HvSdPnz3f7794eW7L2nAx4aUqzUXGrFCyEBOUqMRFZQTTmRLT7OrzWp9+F8bKshjjqhKJZotC5pIz9KW0/4tqWaQNzXKYOqrZ9YZHDmhuGIcmdmOgttZpgyexuxxTJXKcAQyhoUbD2Djoui8pltXf5HBHHm3LI3cAm3PbdOVSdIfHLV97bp3bm9644GB4w7fKyN0agBq5WGKS9gfRUdQG3IW4gwHp4izt/6TzktdaFMgVs3YWRxUmDTMouRKuR2srKsav2ELMPBZMC5s07QAcvPWVOeSl8atAaKvbHQ3T1q505p2a4dLuauvi/7RZjfmHpJFFVaMo+OagvFaAJaynCXNpBEe18sC4kf6uwJfMDw79zHv+E+LdJ9+F8+FR7Pnr8eD0Y/cde+Q1eUPekZi8J6fkCzkjE8KDk4AHKtDhp3AZluG3jTUMup5X5J8If/wBdgTaAQ==</latexit><latexit sha1_base64="OtQJE5t17cn0q9fPDV2tBUgro+Q=">AAACwHicbZFda9swFIZl76vLPpptl7s5LAwGpcUOhY1BoWw3u8hFB0lTiFwjK3IiKtmedDyaGf3J3Yz9mymO12bpDgiec9736OtklZIWo+h3EN67/+Dho73HvSdPnz3f7794eW7L2nAx4aUqzUXGrFCyEBOUqMRFZQTTmRLT7OrzWp9+F8bKshjjqhKJZotC5pIz9KW0/4tqWaQNzXKYOqrZ9YZHDmhuGIcmdmOgttZpgyexuxxTJXKcAQyhoUbD2Djoui8pltXf5HBHHm3LI3cAm3PbdOVSdIfHLV97bp3bm9644GB4w7fKyN0agBq5WGKS9gfRUdQG3IW4gwHp4izt/6TzktdaFMgVs3YWRxUmDTMouRKuR2srKsav2ELMPBZMC5s07QAcvPWVOeSl8atAaKvbHQ3T1q505p2a4dLuauvi/7RZjfmHpJFFVaMo+OagvFaAJaynCXNpBEe18sC4kf6uwJfMDw79zHv+E+LdJ9+F8+FR7Pnr8eD0Y/cde+Q1eUPekZi8J6fkCzkjE8KDk4AHKtDhp3AZluG3jTUMup5X5J8If/wBdgTaAQ==</latexit><latexit sha1_base64="OtQJE5t17cn0q9fPDV2tBUgro+Q=">AAACwHicbZFda9swFIZl76vLPpptl7s5LAwGpcUOhY1BoWw3u8hFB0lTiFwjK3IiKtmedDyaGf3J3Yz9mymO12bpDgiec9736OtklZIWo+h3EN67/+Dho73HvSdPnz3f7794eW7L2nAx4aUqzUXGrFCyEBOUqMRFZQTTmRLT7OrzWp9+F8bKshjjqhKJZotC5pIz9KW0/4tqWaQNzXKYOqrZ9YZHDmhuGIcmdmOgttZpgyexuxxTJXKcAQyhoUbD2Djoui8pltXf5HBHHm3LI3cAm3PbdOVSdIfHLV97bp3bm9644GB4w7fKyN0agBq5WGKS9gfRUdQG3IW4gwHp4izt/6TzktdaFMgVs3YWRxUmDTMouRKuR2srKsav2ELMPBZMC5s07QAcvPWVOeSl8atAaKvbHQ3T1q505p2a4dLuauvi/7RZjfmHpJFFVaMo+OagvFaAJaynCXNpBEe18sC4kf6uwJfMDw79zHv+E+LdJ9+F8+FR7Pnr8eD0Y/cde+Q1eUPekZi8J6fkCzkjE8KDk4AHKtDhp3AZluG3jTUMup5X5J8If/wBdgTaAQ==</latexit>
Costly!
O(DK)<latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit>
Step 1:
Step 2:
for t = 1, 2, 3,…
yt L
�1Wxt
<latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit>
W (1� ↵t)W + ↵tytx>t
L (1� �t)L+ �tyty>t
<latexit sha1_base64="FxUBIQqxQp0afv0GOk8SvjE9Hck=">AAACf3icbZHBbtNAEIbXLi0lQJvCkcuqEVBUGtmoElRcInHhkEORSFMpDtZ4M05WXXut3TGtZeU1eDBuvAsHNk6CaMJIK336Z+bf3ZmkUNJSEPzy/J0Hu3sP9x+1Hj95enDYPnp2ZXVpBA6EVtpcJ2BRyRwHJEnhdWEQskThMLn5tMgPv6OxUudfqSpwnME0l6kUQE6K2z/qKEn5cM5fRQpTAmP0LT8JzyJQxQxierPOn/K1xBupmsfUwJ2DbxHpgkdRq1H6224J0l+zfmO2VDa8qrVX3O4E3aAJvg3hCjpsFZdx+2c00aLMMCehwNpRGBQ0rsGQFArnrai0WIC4gSmOHOaQoR3Xzfzm/KVTJjzVxp2ceKP+21FDZm2VJa4yA5rZzdxC/F9uVFL6YVzLvCgJc7G8KC0VJ80Xy+ATaVCQqhyAMNK9lYsZGBDkVtZyQwg3v7wNV++6oeMv553ex9U49tkLdsxOWMjesx77zC7ZgAn22zv2Tr23vue/9rt+sCz1vVXPc3Yv/Is/9nu/fQ==</latexit><latexit sha1_base64="FxUBIQqxQp0afv0GOk8SvjE9Hck=">AAACf3icbZHBbtNAEIbXLi0lQJvCkcuqEVBUGtmoElRcInHhkEORSFMpDtZ4M05WXXut3TGtZeU1eDBuvAsHNk6CaMJIK336Z+bf3ZmkUNJSEPzy/J0Hu3sP9x+1Hj95enDYPnp2ZXVpBA6EVtpcJ2BRyRwHJEnhdWEQskThMLn5tMgPv6OxUudfqSpwnME0l6kUQE6K2z/qKEn5cM5fRQpTAmP0LT8JzyJQxQxierPOn/K1xBupmsfUwJ2DbxHpgkdRq1H6224J0l+zfmO2VDa8qrVX3O4E3aAJvg3hCjpsFZdx+2c00aLMMCehwNpRGBQ0rsGQFArnrai0WIC4gSmOHOaQoR3Xzfzm/KVTJjzVxp2ceKP+21FDZm2VJa4yA5rZzdxC/F9uVFL6YVzLvCgJc7G8KC0VJ80Xy+ATaVCQqhyAMNK9lYsZGBDkVtZyQwg3v7wNV++6oeMv553ex9U49tkLdsxOWMjesx77zC7ZgAn22zv2Tr23vue/9rt+sCz1vVXPc3Yv/Is/9nu/fQ==</latexit><latexit sha1_base64="FxUBIQqxQp0afv0GOk8SvjE9Hck=">AAACf3icbZHBbtNAEIbXLi0lQJvCkcuqEVBUGtmoElRcInHhkEORSFMpDtZ4M05WXXut3TGtZeU1eDBuvAsHNk6CaMJIK336Z+bf3ZmkUNJSEPzy/J0Hu3sP9x+1Hj95enDYPnp2ZXVpBA6EVtpcJ2BRyRwHJEnhdWEQskThMLn5tMgPv6OxUudfqSpwnME0l6kUQE6K2z/qKEn5cM5fRQpTAmP0LT8JzyJQxQxierPOn/K1xBupmsfUwJ2DbxHpgkdRq1H6224J0l+zfmO2VDa8qrVX3O4E3aAJvg3hCjpsFZdx+2c00aLMMCehwNpRGBQ0rsGQFArnrai0WIC4gSmOHOaQoR3Xzfzm/KVTJjzVxp2ceKP+21FDZm2VJa4yA5rZzdxC/F9uVFL6YVzLvCgJc7G8KC0VJ80Xy+ATaVCQqhyAMNK9lYsZGBDkVtZyQwg3v7wNV++6oeMv553ex9U49tkLdsxOWMjesx77zC7ZgAn22zv2Tr23vue/9rt+sCz1vVXPc3Yv/Is/9nu/fQ==</latexit><latexit sha1_base64="FxUBIQqxQp0afv0GOk8SvjE9Hck=">AAACf3icbZHBbtNAEIbXLi0lQJvCkcuqEVBUGtmoElRcInHhkEORSFMpDtZ4M05WXXut3TGtZeU1eDBuvAsHNk6CaMJIK336Z+bf3ZmkUNJSEPzy/J0Hu3sP9x+1Hj95enDYPnp2ZXVpBA6EVtpcJ2BRyRwHJEnhdWEQskThMLn5tMgPv6OxUudfqSpwnME0l6kUQE6K2z/qKEn5cM5fRQpTAmP0LT8JzyJQxQxierPOn/K1xBupmsfUwJ2DbxHpgkdRq1H6224J0l+zfmO2VDa8qrVX3O4E3aAJvg3hCjpsFZdx+2c00aLMMCehwNpRGBQ0rsGQFArnrai0WIC4gSmOHOaQoR3Xzfzm/KVTJjzVxp2ceKP+21FDZm2VJa4yA5rZzdxC/F9uVFL6YVzLvCgJc7G8KC0VJ80Xy+ATaVCQqhyAMNK9lYsZGBDkVtZyQwg3v7wNV++6oeMv553ex9U49tkLdsxOWMjesx77zC7ZgAn22zv2Tr23vue/9rt+sCz1vVXPc3Yv/Is/9nu/fQ==</latexit>
O(DK +K3)<latexit sha1_base64="4frEQKG9tLsg5TTBxkH+EGDxT4w=">AAAB/3icbVDLSgMxFL1TX7W+RgU3boJFqAhlRgUFNwVdCF1YwT6gHUsmzbShmQdJRihjF/6KGxeKuPU33Pk3ZtpZaPVA4HDOvdyT40acSWVZX0Zubn5hcSm/XFhZXVvfMDe3GjKMBaF1EvJQtFwsKWcBrSumOG1FgmLf5bTpDi9Sv3lPhWRhcKtGEXV83A+YxwhWWuqaOx0fqwHBPLkeo9JlFR2i6t3xQdcsWmVrAvSX2BkpQoZa1/zs9EIS+zRQhGMp27YVKSfBQjHC6bjQiSWNMBniPm1rGmCfSieZ5B+jfa30kBcK/QKFJurPjQT7Uo58V0+maeWsl4r/ee1YeWdOwoIoVjQg00NezJEKUVoG6jFBieIjTTARTGdFZIAFJkpXVtAl2LNf/ksaR2Vb85uTYuU8qyMPu7AHJbDhFCpwBTWoA4EHeIIXeDUejWfjzXifjuaMbGcbfsH4+AZDMJRE</latexit><latexit sha1_base64="4frEQKG9tLsg5TTBxkH+EGDxT4w=">AAAB/3icbVDLSgMxFL1TX7W+RgU3boJFqAhlRgUFNwVdCF1YwT6gHUsmzbShmQdJRihjF/6KGxeKuPU33Pk3ZtpZaPVA4HDOvdyT40acSWVZX0Zubn5hcSm/XFhZXVvfMDe3GjKMBaF1EvJQtFwsKWcBrSumOG1FgmLf5bTpDi9Sv3lPhWRhcKtGEXV83A+YxwhWWuqaOx0fqwHBPLkeo9JlFR2i6t3xQdcsWmVrAvSX2BkpQoZa1/zs9EIS+zRQhGMp27YVKSfBQjHC6bjQiSWNMBniPm1rGmCfSieZ5B+jfa30kBcK/QKFJurPjQT7Uo58V0+maeWsl4r/ee1YeWdOwoIoVjQg00NezJEKUVoG6jFBieIjTTARTGdFZIAFJkpXVtAl2LNf/ksaR2Vb85uTYuU8qyMPu7AHJbDhFCpwBTWoA4EHeIIXeDUejWfjzXifjuaMbGcbfsH4+AZDMJRE</latexit><latexit sha1_base64="4frEQKG9tLsg5TTBxkH+EGDxT4w=">AAAB/3icbVDLSgMxFL1TX7W+RgU3boJFqAhlRgUFNwVdCF1YwT6gHUsmzbShmQdJRihjF/6KGxeKuPU33Pk3ZtpZaPVA4HDOvdyT40acSWVZX0Zubn5hcSm/XFhZXVvfMDe3GjKMBaF1EvJQtFwsKWcBrSumOG1FgmLf5bTpDi9Sv3lPhWRhcKtGEXV83A+YxwhWWuqaOx0fqwHBPLkeo9JlFR2i6t3xQdcsWmVrAvSX2BkpQoZa1/zs9EIS+zRQhGMp27YVKSfBQjHC6bjQiSWNMBniPm1rGmCfSieZ5B+jfa30kBcK/QKFJurPjQT7Uo58V0+maeWsl4r/ee1YeWdOwoIoVjQg00NezJEKUVoG6jFBieIjTTARTGdFZIAFJkpXVtAl2LNf/ksaR2Vb85uTYuU8qyMPu7AHJbDhFCpwBTWoA4EHeIIXeDUejWfjzXifjuaMbGcbfsH4+AZDMJRE</latexit><latexit sha1_base64="4frEQKG9tLsg5TTBxkH+EGDxT4w=">AAAB/3icbVDLSgMxFL1TX7W+RgU3boJFqAhlRgUFNwVdCF1YwT6gHUsmzbShmQdJRihjF/6KGxeKuPU33Pk3ZtpZaPVA4HDOvdyT40acSWVZX0Zubn5hcSm/XFhZXVvfMDe3GjKMBaF1EvJQtFwsKWcBrSumOG1FgmLf5bTpDi9Sv3lPhWRhcKtGEXV83A+YxwhWWuqaOx0fqwHBPLkeo9JlFR2i6t3xQdcsWmVrAvSX2BkpQoZa1/zs9EIS+zRQhGMp27YVKSfBQjHC6bjQiSWNMBniPm1rGmCfSieZ5B+jfa30kBcK/QKFJurPjQT7Uo58V0+maeWsl4r/ee1YeWdOwoIoVjQg00NezJEKUVoG6jFBieIjTTARTGdFZIAFJkpXVtAl2LNf/ksaR2Vb85uTYuU8qyMPu7AHJbDhFCpwBTWoA4EHeIIXeDUejWfjzXifjuaMbGcbfsH4+AZDMJRE</latexit>
Step 1:
Step 2:
for t = 1, 2, 3,…
yt L
�1Wxt
<latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit><latexit sha1_base64="EWJcVjTYcwFj1Ot96Q1a1R4sbTg=">AAACG3icbVDLSgMxFM34rPVVdekmWAQ3lpkiKLgpuHHhooJ9QGccMmmmDc08SO6oZZj/cOOvuHGhiCvBhX9jOp2Fth4IOTn3XG7u8WLBFZjmt7GwuLS8slpaK69vbG5tV3Z22ypKJGUtGolIdj2imOAhawEHwbqxZCTwBOt4o4tJvXPHpOJReAPjmDkBGYTc55SAltxKPbU9H48zF7AtmA9Eyuge5+JVdpseW9n00cny60Eb3UrVrJk58DyxClJFBZpu5dPuRzQJWAhUEKV6lhmDkxIJnAqWle1EsZjQERmwnqYhCZhy0ny3DB9qpY/9SOoTAs7V3x0pCZQaB552BgSGarY2Ef+r9RLwz5yUh3ECLKTTQX4iMER4EhTuc8koiLEmhEqu/4rpkEhCQcdZ1iFYsyvPk3a9Zml+fVJtnBdxlNA+OkBHyEKnqIEuURO1EEWP6Bm9ojfjyXgx3o2PqXXBKHr20B8YXz99cKEI</latexit>
W (1� ↵t)W + ↵tytx>t
L
�1 1
1� �t
✓L
�1 � ctL�1yty
>t L
�1
1 + cty>t L
�1yt
◆
<latexit sha1_base64="6oE/Cje30jomeWT0rfkSMfPY1XE=">AAAC5HicbVJNb9NAEF27fJTw0VCOXFZEoFRVKhshgcSlEhcOHIpEmkrZ1Bpv1smqa6+1OwYsy2cuHECIKz+KW39LL2wcByVpR1rp6b03b+zZjXMlLQbBpefv3Lp95+7uvc79Bw8f7XUf759aXRguhlwrbc5isELJTAxRohJnuRGQxkqM4ot3C330WRgrdfYJy1xMUphlMpEc0FFR96picUJHNX3BlEgQjNFfaD8cMFD5HCI8WOmHdEXRhirrCBvw1YFzhjqnjHWW2of6vBqEm5ksMcCrsK5cdizQ5dS0kfvrLYOlja+mtPTmxPL/xHWPSz7k244bQ2pm5GyOB1G3FxwFTdHrIGxBj7R1EnX/sqnmRSoy5AqsHYdBjpMKDEquRN1hhRU58AuYibGDGaTCTqrmkmr63DFTmmjjToa0Ydc7KkitLdPYOVPAud3WFuRN2rjA5M2kklleoMj4clBSKIqaLm6cTqURHFXpAHAj3bdSPge3ZnTvouOWEG7/8nVw+vIodPjjq97x23Ydu+QpeUb6JCSvyTF5T07IkHAPvG/eD++nn/jf/V/+76XV99qeJ2Sj/D//ALaN6Eo=</latexit><latexit sha1_base64="6oE/Cje30jomeWT0rfkSMfPY1XE=">AAAC5HicbVJNb9NAEF27fJTw0VCOXFZEoFRVKhshgcSlEhcOHIpEmkrZ1Bpv1smqa6+1OwYsy2cuHECIKz+KW39LL2wcByVpR1rp6b03b+zZjXMlLQbBpefv3Lp95+7uvc79Bw8f7XUf759aXRguhlwrbc5isELJTAxRohJnuRGQxkqM4ot3C330WRgrdfYJy1xMUphlMpEc0FFR96picUJHNX3BlEgQjNFfaD8cMFD5HCI8WOmHdEXRhirrCBvw1YFzhjqnjHWW2of6vBqEm5ksMcCrsK5cdizQ5dS0kfvrLYOlja+mtPTmxPL/xHWPSz7k244bQ2pm5GyOB1G3FxwFTdHrIGxBj7R1EnX/sqnmRSoy5AqsHYdBjpMKDEquRN1hhRU58AuYibGDGaTCTqrmkmr63DFTmmjjToa0Ydc7KkitLdPYOVPAud3WFuRN2rjA5M2kklleoMj4clBSKIqaLm6cTqURHFXpAHAj3bdSPge3ZnTvouOWEG7/8nVw+vIodPjjq97x23Ydu+QpeUb6JCSvyTF5T07IkHAPvG/eD++nn/jf/V/+76XV99qeJ2Sj/D//ALaN6Eo=</latexit><latexit sha1_base64="6oE/Cje30jomeWT0rfkSMfPY1XE=">AAAC5HicbVJNb9NAEF27fJTw0VCOXFZEoFRVKhshgcSlEhcOHIpEmkrZ1Bpv1smqa6+1OwYsy2cuHECIKz+KW39LL2wcByVpR1rp6b03b+zZjXMlLQbBpefv3Lp95+7uvc79Bw8f7XUf759aXRguhlwrbc5isELJTAxRohJnuRGQxkqM4ot3C330WRgrdfYJy1xMUphlMpEc0FFR96picUJHNX3BlEgQjNFfaD8cMFD5HCI8WOmHdEXRhirrCBvw1YFzhjqnjHWW2of6vBqEm5ksMcCrsK5cdizQ5dS0kfvrLYOlja+mtPTmxPL/xHWPSz7k244bQ2pm5GyOB1G3FxwFTdHrIGxBj7R1EnX/sqnmRSoy5AqsHYdBjpMKDEquRN1hhRU58AuYibGDGaTCTqrmkmr63DFTmmjjToa0Ydc7KkitLdPYOVPAud3WFuRN2rjA5M2kklleoMj4clBSKIqaLm6cTqURHFXpAHAj3bdSPge3ZnTvouOWEG7/8nVw+vIodPjjq97x23Ydu+QpeUb6JCSvyTF5T07IkHAPvG/eD++nn/jf/V/+76XV99qeJ2Sj/D//ALaN6Eo=</latexit><latexit sha1_base64="6oE/Cje30jomeWT0rfkSMfPY1XE=">AAAC5HicbVJNb9NAEF27fJTw0VCOXFZEoFRVKhshgcSlEhcOHIpEmkrZ1Bpv1smqa6+1OwYsy2cuHECIKz+KW39LL2wcByVpR1rp6b03b+zZjXMlLQbBpefv3Lp95+7uvc79Bw8f7XUf759aXRguhlwrbc5isELJTAxRohJnuRGQxkqM4ot3C330WRgrdfYJy1xMUphlMpEc0FFR96picUJHNX3BlEgQjNFfaD8cMFD5HCI8WOmHdEXRhirrCBvw1YFzhjqnjHWW2of6vBqEm5ksMcCrsK5cdizQ5dS0kfvrLYOlja+mtPTmxPL/xHWPSz7k244bQ2pm5GyOB1G3FxwFTdHrIGxBj7R1EnX/sqnmRSoy5AqsHYdBjpMKDEquRN1hhRU58AuYibGDGaTCTqrmkmr63DFTmmjjToa0Ydc7KkitLdPYOVPAud3WFuRN2rjA5M2kklleoMj4clBSKIqaLm6cTqURHFXpAHAj3bdSPge3ZnTvouOWEG7/8nVw+vIodPjjq97x23Ydu+QpeUb6JCSvyTF5T07IkHAPvG/eD++nn/jf/V/+76XV99qeJ2Sj/D//ALaN6Eo=</latexit>
O(DK)<latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit><latexit sha1_base64="yiLz7FLLmI6tMMtPdGG9v8PXU0k=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyWRgoKbgi4EF1awD2hDmUwn7dDJJMxMCiX0T9y4UMStf+LOv3HSZqGtBwYO59zLPXP8mDOlHefbKqytb2xuFbdLO7t7+wf24VFLRYkktEkiHsmOjxXlTNCmZprTTiwpDn1O2/74JvPbEyoVi8STnsbUC/FQsIARrI3Ut+1eiPWIYJ4+zFDl9v68b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWhdV1/DHWrl+nddRhBM4hQq4cAl1uIMGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx9QpJLB</latexit>
Fast Similarity Matching: store and use Sherman-Morrison formula to update itL�1<latexit sha1_base64="vNYgNCBxetcZDeam3MyDb6c0dvw=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDgpuDGhYsK9gLttGTSTBuaSYYko5Rh3sONC0Xc+i7ufBvTdhba+kPg4z/ncE7+IOZMG9f9dgorq2vrG8XN0tb2zu5eef+gqWWiCG0QyaVqB1hTzgRtGGY4bceK4ijgtBWMb6b11iNVmknxYCYx9SM8FCxkBBtr9VDaDUJ0l/XSMy9D/XLFrbozoWXwcqhArnq//NUdSJJEVBjCsdYdz42Nn2JlGOE0K3UTTWNMxnhIOxYFjqj209nVGTqxzgCFUtknDJq5vydSHGk9iQLbGWEz0ou1qflfrZOY8MpPmYgTQwWZLwoTjoxE0wjQgClKDJ9YwEQxeysiI6wwMTaokg3BW/zyMjTPq57l+4tK7TqPowhHcAyn4MEl1OAW6tAAAgqe4RXenCfnxXl3PuatBSefOYQ/cj5/APjEkXs=</latexit><latexit sha1_base64="vNYgNCBxetcZDeam3MyDb6c0dvw=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDgpuDGhYsK9gLttGTSTBuaSYYko5Rh3sONC0Xc+i7ufBvTdhba+kPg4z/ncE7+IOZMG9f9dgorq2vrG8XN0tb2zu5eef+gqWWiCG0QyaVqB1hTzgRtGGY4bceK4ijgtBWMb6b11iNVmknxYCYx9SM8FCxkBBtr9VDaDUJ0l/XSMy9D/XLFrbozoWXwcqhArnq//NUdSJJEVBjCsdYdz42Nn2JlGOE0K3UTTWNMxnhIOxYFjqj209nVGTqxzgCFUtknDJq5vydSHGk9iQLbGWEz0ou1qflfrZOY8MpPmYgTQwWZLwoTjoxE0wjQgClKDJ9YwEQxeysiI6wwMTaokg3BW/zyMjTPq57l+4tK7TqPowhHcAyn4MEl1OAW6tAAAgqe4RXenCfnxXl3PuatBSefOYQ/cj5/APjEkXs=</latexit><latexit sha1_base64="vNYgNCBxetcZDeam3MyDb6c0dvw=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDgpuDGhYsK9gLttGTSTBuaSYYko5Rh3sONC0Xc+i7ufBvTdhba+kPg4z/ncE7+IOZMG9f9dgorq2vrG8XN0tb2zu5eef+gqWWiCG0QyaVqB1hTzgRtGGY4bceK4ijgtBWMb6b11iNVmknxYCYx9SM8FCxkBBtr9VDaDUJ0l/XSMy9D/XLFrbozoWXwcqhArnq//NUdSJJEVBjCsdYdz42Nn2JlGOE0K3UTTWNMxnhIOxYFjqj209nVGTqxzgCFUtknDJq5vydSHGk9iQLbGWEz0ou1qflfrZOY8MpPmYgTQwWZLwoTjoxE0wjQgClKDJ9YwEQxeysiI6wwMTaokg3BW/zyMjTPq57l+4tK7TqPowhHcAyn4MEl1OAW6tAAAgqe4RXenCfnxXl3PuatBSefOYQ/cj5/APjEkXs=</latexit>
O(K2)<latexit sha1_base64="6iRL9RFxkE5nYSf24qJzJlX6u5Q=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyUpgoKbghvBhRXsA9pYJtNJO3QyCTOTQgn9EzcuFHHrn7jzb5y0WWjrgYHDOfdyzxw/5kxpx/m2CmvrG5tbxe3Szu7e/oF9eNRSUSIJbZKIR7LjY0U5E7Spmea0E0uKQ5/Ttj++yfz2hErFIvGopzH1QjwULGAEayP1bbsXYj0imKf3s8rdU+28b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWrWqa/jDRbl+nddRhBM4hQq4cAl1uIUGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx+T+JLt</latexit><latexit sha1_base64="6iRL9RFxkE5nYSf24qJzJlX6u5Q=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyUpgoKbghvBhRXsA9pYJtNJO3QyCTOTQgn9EzcuFHHrn7jzb5y0WWjrgYHDOfdyzxw/5kxpx/m2CmvrG5tbxe3Szu7e/oF9eNRSUSIJbZKIR7LjY0U5E7Spmea0E0uKQ5/Ttj++yfz2hErFIvGopzH1QjwULGAEayP1bbsXYj0imKf3s8rdU+28b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWrWqa/jDRbl+nddRhBM4hQq4cAl1uIUGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx+T+JLt</latexit><latexit sha1_base64="6iRL9RFxkE5nYSf24qJzJlX6u5Q=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyUpgoKbghvBhRXsA9pYJtNJO3QyCTOTQgn9EzcuFHHrn7jzb5y0WWjrgYHDOfdyzxw/5kxpx/m2CmvrG5tbxe3Szu7e/oF9eNRSUSIJbZKIR7LjY0U5E7Spmea0E0uKQ5/Ttj++yfz2hErFIvGopzH1QjwULGAEayP1bbsXYj0imKf3s8rdU+28b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWrWqa/jDRbl+nddRhBM4hQq4cAl1uIUGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx+T+JLt</latexit><latexit sha1_base64="6iRL9RFxkE5nYSf24qJzJlX6u5Q=">AAAB+XicbVDLSsNAFL2pr1pfUZduBotQNyUpgoKbghvBhRXsA9pYJtNJO3QyCTOTQgn9EzcuFHHrn7jzb5y0WWjrgYHDOfdyzxw/5kxpx/m2CmvrG5tbxe3Szu7e/oF9eNRSUSIJbZKIR7LjY0U5E7Spmea0E0uKQ5/Ttj++yfz2hErFIvGopzH1QjwULGAEayP1bbsXYj0imKf3s8rdU+28b5edqjMHWiVuTsqQo9G3v3qDiCQhFZpwrFTXdWLtpVhqRjidlXqJojEmYzykXUMFDqny0nnyGTozygAFkTRPaDRXf2+kOFRqGvpmMsuplr1M/M/rJjq48lIm4kRTQRaHgoQjHaGsBjRgkhLNp4ZgIpnJisgIS0y0KatkSnCXv7xKWrWqa/jDRbl+nddRhBM4hQq4cAl1uIUGNIHABJ7hFd6s1Hqx3q2PxWjByneO4Q+szx+T+JLt</latexit>
Fast Similarity-Matching for Big Data
Fast Similarity-Matching algorithm is competitive!
Minden, Giovannucci, Pehlevan, Chkvlovskii, IEEE Big Data, 2018
Python and MATLAB implementations in Flatiron GitHub
Optimal memory and computational complexity O(nk)<latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit><latexit sha1_base64="(null)">(null)</latexit>
Strategy for deriving NN algorithms from similarity-based cost functions
min8t,yt2⌦
"�
TX
t=1
TX
t0=1
y
>t yt0x
>t xt0 + f(y1, ...,yt)
#
<latexit sha1_base64="NFWdC4WStsA/7HMscnFKPFqb0nk=">AAACkXicbVFdb9MwFHXC1yiwFfbIi0WFNkSJEjQJkBiq4AWJB4a0bpPqLHLcm9aa7UT2DVoV5f/we3jj3+A23VQ2rmTp3HPuta/PzSslHcbxnyC8c/fe/QdbD3uPHj/Z3uk/fXbiytoKGItSlfYs5w6UNDBGiQrOKgtc5wpO84svS/30J1gnS3OMiwpSzWdGFlJw9FTW/0Up09JkDStKy5WiOKQNywu6aDNk0lD2XcOMt5QpKHDyhjJX66zBw6Q9P75K9rrsurHB9pxhWW0Qe22XXPprN7TLTntNi/2r2mQYRdHweoZXzMrZHNOsP4ijeBX0NkjWYEDWcZT1f7NpKWoNBoXizk2SuMK04RalUND2WO2g4uKCz2DioeEaXNqsHG3pS89MqbfEH4N0xW52NFw7t9C5r9Qc5+6mtiT/p01qLN6njTRVjWBE91BRe9tLulwPnUoLAtXCAy6s9LNSMeeWC/RL7HkTkptfvg1O3kaJxz8OBqOPazu2yHPyguyThLwjI/KVHJExEcF2cBAcBp/C3fBDOAo/d6VhsO7ZJf9E+O0vgJ/GNQ==</latexit><latexit sha1_base64="NFWdC4WStsA/7HMscnFKPFqb0nk=">AAACkXicbVFdb9MwFHXC1yiwFfbIi0WFNkSJEjQJkBiq4AWJB4a0bpPqLHLcm9aa7UT2DVoV5f/we3jj3+A23VQ2rmTp3HPuta/PzSslHcbxnyC8c/fe/QdbD3uPHj/Z3uk/fXbiytoKGItSlfYs5w6UNDBGiQrOKgtc5wpO84svS/30J1gnS3OMiwpSzWdGFlJw9FTW/0Up09JkDStKy5WiOKQNywu6aDNk0lD2XcOMt5QpKHDyhjJX66zBw6Q9P75K9rrsurHB9pxhWW0Qe22XXPprN7TLTntNi/2r2mQYRdHweoZXzMrZHNOsP4ijeBX0NkjWYEDWcZT1f7NpKWoNBoXizk2SuMK04RalUND2WO2g4uKCz2DioeEaXNqsHG3pS89MqbfEH4N0xW52NFw7t9C5r9Qc5+6mtiT/p01qLN6njTRVjWBE91BRe9tLulwPnUoLAtXCAy6s9LNSMeeWC/RL7HkTkptfvg1O3kaJxz8OBqOPazu2yHPyguyThLwjI/KVHJExEcF2cBAcBp/C3fBDOAo/d6VhsO7ZJf9E+O0vgJ/GNQ==</latexit><latexit sha1_base64="NFWdC4WStsA/7HMscnFKPFqb0nk=">AAACkXicbVFdb9MwFHXC1yiwFfbIi0WFNkSJEjQJkBiq4AWJB4a0bpPqLHLcm9aa7UT2DVoV5f/we3jj3+A23VQ2rmTp3HPuta/PzSslHcbxnyC8c/fe/QdbD3uPHj/Z3uk/fXbiytoKGItSlfYs5w6UNDBGiQrOKgtc5wpO84svS/30J1gnS3OMiwpSzWdGFlJw9FTW/0Up09JkDStKy5WiOKQNywu6aDNk0lD2XcOMt5QpKHDyhjJX66zBw6Q9P75K9rrsurHB9pxhWW0Qe22XXPprN7TLTntNi/2r2mQYRdHweoZXzMrZHNOsP4ijeBX0NkjWYEDWcZT1f7NpKWoNBoXizk2SuMK04RalUND2WO2g4uKCz2DioeEaXNqsHG3pS89MqbfEH4N0xW52NFw7t9C5r9Qc5+6mtiT/p01qLN6njTRVjWBE91BRe9tLulwPnUoLAtXCAy6s9LNSMeeWC/RL7HkTkptfvg1O3kaJxz8OBqOPazu2yHPyguyThLwjI/KVHJExEcF2cBAcBp/C3fBDOAo/d6VhsO7ZJf9E+O0vgJ/GNQ==</latexit><latexit sha1_base64="NFWdC4WStsA/7HMscnFKPFqb0nk=">AAACkXicbVFdb9MwFHXC1yiwFfbIi0WFNkSJEjQJkBiq4AWJB4a0bpPqLHLcm9aa7UT2DVoV5f/we3jj3+A23VQ2rmTp3HPuta/PzSslHcbxnyC8c/fe/QdbD3uPHj/Z3uk/fXbiytoKGItSlfYs5w6UNDBGiQrOKgtc5wpO84svS/30J1gnS3OMiwpSzWdGFlJw9FTW/0Up09JkDStKy5WiOKQNywu6aDNk0lD2XcOMt5QpKHDyhjJX66zBw6Q9P75K9rrsurHB9pxhWW0Qe22XXPprN7TLTntNi/2r2mQYRdHweoZXzMrZHNOsP4ijeBX0NkjWYEDWcZT1f7NpKWoNBoXizk2SuMK04RalUND2WO2g4uKCz2DioeEaXNqsHG3pS89MqbfEH4N0xW52NFw7t9C5r9Qc5+6mtiT/p01qLN6njTRVjWBE91BRe9tLulwPnUoLAtXCAy6s9LNSMeeWC/RL7HkTkptfvg1O3kaJxz8OBqOPazu2yHPyguyThLwjI/KVHJExEcF2cBAcBp/C3fBDOAo/d6VhsO7ZJf9E+O0vgJ/GNQ==</latexit>
Other circuit elements
Stochastic gradient descent
�Wij = ⌘(yixj �Wij)<latexit sha1_base64="tBrhaWTJdu5LP5AQAY8D9mFo1Dg=">AAACD3icbZA9SwNBEIb34nf8OrW0WQxKLJQ7EbRQELSwjGCMkAvH3maia/Y+2J0TjyP/wMa/YmOhiK2tnf/GTXKFJr6w8PDODLPzBokUGh3n2ypNTE5Nz8zOlecXFpeW7ZXVKx2nikOdxzJW1wHTIEUEdRQo4TpRwMJAQiPonvbrjXtQWsTRJWYJtEJ2E4mO4AyN5dtb3hlIZLTh5+KuR4+pB8iqmS/og39Hdwp/27crzq4zEB0Ht4AKKVTz7S+vHfM0hAi5ZFo3XSfBVs4UCi6hV/ZSDQnjXXYDTYMRC0G38sE9PbppnDbtxMq8COnA/T2Rs1DrLAxMZ8jwVo/W+uZ/tWaKncNWLqIkRYj4cFEnlRRj2g+HtoUCjjIzwLgS5q+U3zLFOJoIyyYEd/Tkcbja23UNX+xXTo6KOGbJOtkgVeKSA3JCzkmN1Aknj+SZvJI368l6sd6tj2FrySpm1sgfWZ8/ce+a/A==</latexit><latexit sha1_base64="tBrhaWTJdu5LP5AQAY8D9mFo1Dg=">AAACD3icbZA9SwNBEIb34nf8OrW0WQxKLJQ7EbRQELSwjGCMkAvH3maia/Y+2J0TjyP/wMa/YmOhiK2tnf/GTXKFJr6w8PDODLPzBokUGh3n2ypNTE5Nz8zOlecXFpeW7ZXVKx2nikOdxzJW1wHTIEUEdRQo4TpRwMJAQiPonvbrjXtQWsTRJWYJtEJ2E4mO4AyN5dtb3hlIZLTh5+KuR4+pB8iqmS/og39Hdwp/27crzq4zEB0Ht4AKKVTz7S+vHfM0hAi5ZFo3XSfBVs4UCi6hV/ZSDQnjXXYDTYMRC0G38sE9PbppnDbtxMq8COnA/T2Rs1DrLAxMZ8jwVo/W+uZ/tWaKncNWLqIkRYj4cFEnlRRj2g+HtoUCjjIzwLgS5q+U3zLFOJoIyyYEd/Tkcbja23UNX+xXTo6KOGbJOtkgVeKSA3JCzkmN1Aknj+SZvJI368l6sd6tj2FrySpm1sgfWZ8/ce+a/A==</latexit><latexit sha1_base64="tBrhaWTJdu5LP5AQAY8D9mFo1Dg=">AAACD3icbZA9SwNBEIb34nf8OrW0WQxKLJQ7EbRQELSwjGCMkAvH3maia/Y+2J0TjyP/wMa/YmOhiK2tnf/GTXKFJr6w8PDODLPzBokUGh3n2ypNTE5Nz8zOlecXFpeW7ZXVKx2nikOdxzJW1wHTIEUEdRQo4TpRwMJAQiPonvbrjXtQWsTRJWYJtEJ2E4mO4AyN5dtb3hlIZLTh5+KuR4+pB8iqmS/og39Hdwp/27crzq4zEB0Ht4AKKVTz7S+vHfM0hAi5ZFo3XSfBVs4UCi6hV/ZSDQnjXXYDTYMRC0G38sE9PbppnDbtxMq8COnA/T2Rs1DrLAxMZ8jwVo/W+uZ/tWaKncNWLqIkRYj4cFEnlRRj2g+HtoUCjjIzwLgS5q+U3zLFOJoIyyYEd/Tkcbja23UNX+xXTo6KOGbJOtkgVeKSA3JCzkmN1Aknj+SZvJI368l6sd6tj2FrySpm1sgfWZ8/ce+a/A==</latexit><latexit sha1_base64="tBrhaWTJdu5LP5AQAY8D9mFo1Dg=">AAACD3icbZA9SwNBEIb34nf8OrW0WQxKLJQ7EbRQELSwjGCMkAvH3maia/Y+2J0TjyP/wMa/YmOhiK2tnf/GTXKFJr6w8PDODLPzBokUGh3n2ypNTE5Nz8zOlecXFpeW7ZXVKx2nikOdxzJW1wHTIEUEdRQo4TpRwMJAQiPonvbrjXtQWsTRJWYJtEJ2E4mO4AyN5dtb3hlIZLTh5+KuR4+pB8iqmS/og39Hdwp/27crzq4zEB0Ht4AKKVTz7S+vHfM0hAi5ZFo3XSfBVs4UCi6hV/ZSDQnjXXYDTYMRC0G38sE9PbppnDbtxMq8COnA/T2Rs1DrLAxMZ8jwVo/W+uZ/tWaKncNWLqIkRYj4cFEnlRRj2g+HtoUCjjIzwLgS5q+U3zLFOJoIyyYEd/Tkcbja23UNX+xXTo6KOGbJOtkgVeKSA3JCzkmN1Aknj+SZvJI368l6sd6tj2FrySpm1sgfWZ8/ce+a/A==</latexit>
A growing bio-math dictionary
Nonnegative similarity-matching (NSM) network
min8t,yt�0
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0 � ↵
�2
<latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit>
x1
xn
. . .
y1
y2x2
yk
. . .
. . .Inhibitory Hebbian (anti-Hebbian)
Hebbian Wij �Wij + ⌘ (yixj �Wij)
Lij � Lij + ⌘ (yiyj � Lij)
b � b+ ⌘ (↵y � b)<latexit sha1_base64="Pmai8XYZRTGQZTheeqjxRc8s5qI=">AAACMHicbZDPShxBEMZ7jIm6+eNGj7k0WRYMITIjgnoTPMSjgawKO8tS01uz09jTPXTXJAzDPlIuPoq5RFAk1zyFPesEopuChh/fV0VXfUmhpKMwvA6Wni0/f7GyutZ5+er1m/Xu241TZ0orcCCMMvY8AYdKahyQJIXnhUXIE4VnycVR4599Q+uk0V+pKnCUw1TLVAogL427n+s4SXky47EyeqowJbDWfOd/5Y88RoK4MbZiUEUGc6eafWo7YiunGX0Yd3vhdjgvvghRCz3W1sm4exVPjChz1CQUODeMwoJGNViSQuGsE5cOCxAXMMWhRw05ulE9P3jG+16Z8NRY/zTxufrvRA25c1Xu9+/nQJl76jXi/7xhSen+qJa6KAm1ePgoLRUnw5v0+ERaFKQqDyCs9LtykYEFQT7jjg8henryIpzubEeev+z2Dg/aOFbZO/aebbGI7bFDdsxO2IAJ9oP9ZDfsNrgMfgV3we+H1qWgndlkjyr4cw/vl6l7</latexit><latexit sha1_base64="Pmai8XYZRTGQZTheeqjxRc8s5qI=">AAACMHicbZDPShxBEMZ7jIm6+eNGj7k0WRYMITIjgnoTPMSjgawKO8tS01uz09jTPXTXJAzDPlIuPoq5RFAk1zyFPesEopuChh/fV0VXfUmhpKMwvA6Wni0/f7GyutZ5+er1m/Xu241TZ0orcCCMMvY8AYdKahyQJIXnhUXIE4VnycVR4599Q+uk0V+pKnCUw1TLVAogL427n+s4SXky47EyeqowJbDWfOd/5Y88RoK4MbZiUEUGc6eafWo7YiunGX0Yd3vhdjgvvghRCz3W1sm4exVPjChz1CQUODeMwoJGNViSQuGsE5cOCxAXMMWhRw05ulE9P3jG+16Z8NRY/zTxufrvRA25c1Xu9+/nQJl76jXi/7xhSen+qJa6KAm1ePgoLRUnw5v0+ERaFKQqDyCs9LtykYEFQT7jjg8henryIpzubEeev+z2Dg/aOFbZO/aebbGI7bFDdsxO2IAJ9oP9ZDfsNrgMfgV3we+H1qWgndlkjyr4cw/vl6l7</latexit><latexit sha1_base64="Pmai8XYZRTGQZTheeqjxRc8s5qI=">AAACMHicbZDPShxBEMZ7jIm6+eNGj7k0WRYMITIjgnoTPMSjgawKO8tS01uz09jTPXTXJAzDPlIuPoq5RFAk1zyFPesEopuChh/fV0VXfUmhpKMwvA6Wni0/f7GyutZ5+er1m/Xu241TZ0orcCCMMvY8AYdKahyQJIXnhUXIE4VnycVR4599Q+uk0V+pKnCUw1TLVAogL427n+s4SXky47EyeqowJbDWfOd/5Y88RoK4MbZiUEUGc6eafWo7YiunGX0Yd3vhdjgvvghRCz3W1sm4exVPjChz1CQUODeMwoJGNViSQuGsE5cOCxAXMMWhRw05ulE9P3jG+16Z8NRY/zTxufrvRA25c1Xu9+/nQJl76jXi/7xhSen+qJa6KAm1ePgoLRUnw5v0+ERaFKQqDyCs9LtykYEFQT7jjg8henryIpzubEeev+z2Dg/aOFbZO/aebbGI7bFDdsxO2IAJ9oP9ZDfsNrgMfgV3we+H1qWgndlkjyr4cw/vl6l7</latexit><latexit sha1_base64="Pmai8XYZRTGQZTheeqjxRc8s5qI=">AAACMHicbZDPShxBEMZ7jIm6+eNGj7k0WRYMITIjgnoTPMSjgawKO8tS01uz09jTPXTXJAzDPlIuPoq5RFAk1zyFPesEopuChh/fV0VXfUmhpKMwvA6Wni0/f7GyutZ5+er1m/Xu241TZ0orcCCMMvY8AYdKahyQJIXnhUXIE4VnycVR4599Q+uk0V+pKnCUw1TLVAogL427n+s4SXky47EyeqowJbDWfOd/5Y88RoK4MbZiUEUGc6eafWo7YiunGX0Yd3vhdjgvvghRCz3W1sm4exVPjChz1CQUODeMwoJGNViSQuGsE5cOCxAXMMWhRw05ulE9P3jG+16Z8NRY/zTxufrvRA25c1Xu9+/nQJl76jXi/7xhSen+qJa6KAm1ePgoLRUnw5v0+ERaFKQqDyCs9LtykYEFQT7jjg8henryIpzubEeev+z2Dg/aOFbZO/aebbGI7bFDdsxO2IAJ9oP9ZDfsNrgMfgV3we+H1qWgndlkjyr4cw/vl6l7</latexit>
Homeostatic plasticity
(Földiák, 1990; Rubner & Schulten, 1990; Pehlevan & Chklovskii, 2014; Seung & Zhung, 2017)
˙
y = �y +max (Wx� Ly � b, 0)<latexit sha1_base64="2Pe8pdpza3Ob7LTqaHvXdt2SV+w=">AAACOnicbVDLSsNAFJ34tr6iLt0MFqHig0QEdSEU3LhwUcFaoSllMp20QycPZm7EEPJdbvwKdy7cuFDErR/gNImg1QPDnDn3XObe40aCK7CsJ2Nicmp6ZnZuvrKwuLS8Yq6uXaswlpQ1aShCeeMSxQQPWBM4CHYTSUZ8V7CWOzwb1Vu3TCoeBleQRKzjk37APU4JaKlrXjq9EFLH9XCS4VO89013sOOTO0cwD2q51sry6y4rLBdZ6SyebrZrOZL3B7DdNavWvpUD/yV2SaqoRKNrPuoZaOyzAKggSrVtK4JOSiRwKlhWcWLFIkKHpM/amgbEZ6qT5qtneEsrPeyFUp8AcK7+7EiJr1Tiu9rpExio8dpI/K/WjsE77qQ8iGJgAS0+8mKBIcSjHHGPS0ZBJJoQKrmeFdMBkYSCTruiQ7DHV/5Lrg/2bc0vD6v1kzKOObSBNlEN2egI1dE5aqAmougePaNX9GY8GC/Gu/FRWCeMsmcd/YLx+QXUXKw6</latexit><latexit sha1_base64="2Pe8pdpza3Ob7LTqaHvXdt2SV+w=">AAACOnicbVDLSsNAFJ34tr6iLt0MFqHig0QEdSEU3LhwUcFaoSllMp20QycPZm7EEPJdbvwKdy7cuFDErR/gNImg1QPDnDn3XObe40aCK7CsJ2Nicmp6ZnZuvrKwuLS8Yq6uXaswlpQ1aShCeeMSxQQPWBM4CHYTSUZ8V7CWOzwb1Vu3TCoeBleQRKzjk37APU4JaKlrXjq9EFLH9XCS4VO89013sOOTO0cwD2q51sry6y4rLBdZ6SyebrZrOZL3B7DdNavWvpUD/yV2SaqoRKNrPuoZaOyzAKggSrVtK4JOSiRwKlhWcWLFIkKHpM/amgbEZ6qT5qtneEsrPeyFUp8AcK7+7EiJr1Tiu9rpExio8dpI/K/WjsE77qQ8iGJgAS0+8mKBIcSjHHGPS0ZBJJoQKrmeFdMBkYSCTruiQ7DHV/5Lrg/2bc0vD6v1kzKOObSBNlEN2egI1dE5aqAmougePaNX9GY8GC/Gu/FRWCeMsmcd/YLx+QXUXKw6</latexit><latexit sha1_base64="2Pe8pdpza3Ob7LTqaHvXdt2SV+w=">AAACOnicbVDLSsNAFJ34tr6iLt0MFqHig0QEdSEU3LhwUcFaoSllMp20QycPZm7EEPJdbvwKdy7cuFDErR/gNImg1QPDnDn3XObe40aCK7CsJ2Nicmp6ZnZuvrKwuLS8Yq6uXaswlpQ1aShCeeMSxQQPWBM4CHYTSUZ8V7CWOzwb1Vu3TCoeBleQRKzjk37APU4JaKlrXjq9EFLH9XCS4VO89013sOOTO0cwD2q51sry6y4rLBdZ6SyebrZrOZL3B7DdNavWvpUD/yV2SaqoRKNrPuoZaOyzAKggSrVtK4JOSiRwKlhWcWLFIkKHpM/amgbEZ6qT5qtneEsrPeyFUp8AcK7+7EiJr1Tiu9rpExio8dpI/K/WjsE77qQ8iGJgAS0+8mKBIcSjHHGPS0ZBJJoQKrmeFdMBkYSCTruiQ7DHV/5Lrg/2bc0vD6v1kzKOObSBNlEN2egI1dE5aqAmougePaNX9GY8GC/Gu/FRWCeMsmcd/YLx+QXUXKw6</latexit><latexit sha1_base64="2Pe8pdpza3Ob7LTqaHvXdt2SV+w=">AAACOnicbVDLSsNAFJ34tr6iLt0MFqHig0QEdSEU3LhwUcFaoSllMp20QycPZm7EEPJdbvwKdy7cuFDErR/gNImg1QPDnDn3XObe40aCK7CsJ2Nicmp6ZnZuvrKwuLS8Yq6uXaswlpQ1aShCeeMSxQQPWBM4CHYTSUZ8V7CWOzwb1Vu3TCoeBleQRKzjk37APU4JaKlrXjq9EFLH9XCS4VO89013sOOTO0cwD2q51sry6y4rLBdZ6SyebrZrOZL3B7DdNavWvpUD/yV2SaqoRKNrPuoZaOyzAKggSrVtK4JOSiRwKlhWcWLFIkKHpM/amgbEZ6qT5qtneEsrPeyFUp8AcK7+7EiJr1Tiu9rpExio8dpI/K/WjsE77qQ8iGJgAS0+8mKBIcSjHHGPS0ZBJJoQKrmeFdMBkYSCTruiQ7DHV/5Lrg/2bc0vD6v1kzKOObSBNlEN2egI1dE5aqAmougePaNX9GY8GC/Gu/FRWCeMsmcd/YLx+QXUXKw6</latexit>
Sparse encoding Clustering
(Nonnegative) ICAManifold learning (by tiling)
Spiking implementationwill be on arXiv soon!
Oriented edgereceptive fields
Natural images
x1
xn
. . .
y1
y2x2
yk
. . .
. . .
log 10
Pro
babi
lity
Output Activity0 2 4 6 8 10 12 14 16
-6
-5
-4
-3
-2
-1
0
Sparse activity
Sparse encoding
↵ = 0<latexit sha1_base64="vwY4kkA1z9fcKQdEnly2Rh3T1/w=">AAAB8XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepBoeDFYwX7gW0ok+2mXbrZhN2NUEL/hRcPinj133jz37htc9DWFxYe3plhZ94gEVwb1/12CmvrG5tbxe3Szu7e/kH58Kil41RR1qSxiFUnQM0El6xpuBGskyiGUSBYOxjfzurtJ6Y0j+WDmSTMj3AoecgpGms99lAkIyQ3xO2XK27VnYusgpdDBXI1+uWv3iCmacSkoQK17npuYvwMleFUsGmpl2qWIB3jkHUtSoyY9rP5xlNyZp0BCWNlnzRk7v6eyDDSehIFtjNCM9LLtZn5X62bmvDKz7hMUsMkXXwUpoKYmMzOJwOuGDViYgGp4nZXQkeokBobUsmG4C2fvAqti6pn+f6yUr/O4yjCCZzCOXhQgzrcQQOaQEHCM7zCm6OdF+fd+Vi0Fpx85hj+yPn8AS5Nj+c=</latexit><latexit sha1_base64="vwY4kkA1z9fcKQdEnly2Rh3T1/w=">AAAB8XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepBoeDFYwX7gW0ok+2mXbrZhN2NUEL/hRcPinj133jz37htc9DWFxYe3plhZ94gEVwb1/12CmvrG5tbxe3Szu7e/kH58Kil41RR1qSxiFUnQM0El6xpuBGskyiGUSBYOxjfzurtJ6Y0j+WDmSTMj3AoecgpGms99lAkIyQ3xO2XK27VnYusgpdDBXI1+uWv3iCmacSkoQK17npuYvwMleFUsGmpl2qWIB3jkHUtSoyY9rP5xlNyZp0BCWNlnzRk7v6eyDDSehIFtjNCM9LLtZn5X62bmvDKz7hMUsMkXXwUpoKYmMzOJwOuGDViYgGp4nZXQkeokBobUsmG4C2fvAqti6pn+f6yUr/O4yjCCZzCOXhQgzrcQQOaQEHCM7zCm6OdF+fd+Vi0Fpx85hj+yPn8AS5Nj+c=</latexit><latexit sha1_base64="vwY4kkA1z9fcKQdEnly2Rh3T1/w=">AAAB8XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepBoeDFYwX7gW0ok+2mXbrZhN2NUEL/hRcPinj133jz37htc9DWFxYe3plhZ94gEVwb1/12CmvrG5tbxe3Szu7e/kH58Kil41RR1qSxiFUnQM0El6xpuBGskyiGUSBYOxjfzurtJ6Y0j+WDmSTMj3AoecgpGms99lAkIyQ3xO2XK27VnYusgpdDBXI1+uWv3iCmacSkoQK17npuYvwMleFUsGmpl2qWIB3jkHUtSoyY9rP5xlNyZp0BCWNlnzRk7v6eyDDSehIFtjNCM9LLtZn5X62bmvDKz7hMUsMkXXwUpoKYmMzOJwOuGDViYgGp4nZXQkeokBobUsmG4C2fvAqti6pn+f6yUr/O4yjCCZzCOXhQgzrcQQOaQEHCM7zCm6OdF+fd+Vi0Fpx85hj+yPn8AS5Nj+c=</latexit><latexit sha1_base64="vwY4kkA1z9fcKQdEnly2Rh3T1/w=">AAAB8XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepBoeDFYwX7gW0ok+2mXbrZhN2NUEL/hRcPinj133jz37htc9DWFxYe3plhZ94gEVwb1/12CmvrG5tbxe3Szu7e/kH58Kil41RR1qSxiFUnQM0El6xpuBGskyiGUSBYOxjfzurtJ6Y0j+WDmSTMj3AoecgpGms99lAkIyQ3xO2XK27VnYusgpdDBXI1+uWv3iCmacSkoQK17npuYvwMleFUsGmpl2qWIB3jkHUtSoyY9rP5xlNyZp0BCWNlnzRk7v6eyDDSehIFtjNCM9LLtZn5X62bmvDKz7hMUsMkXXwUpoKYmMzOJwOuGDViYgGp4nZXQkeokBobUsmG4C2fvAqti6pn+f6yUr/O4yjCCZzCOXhQgzrcQQOaQEHCM7zCm6OdF+fd+Vi0Fpx85hj+yPn8AS5Nj+c=</latexit>
Pehlevan & Chklovskii, 2014; also Földiák, 1990; Seung & Zhung, 2017
Convolutional NSM + pooling is a competitive feature extractor!
Inputs are linear mixtures of independent, nonnegative sources.
Task: Given mixtures, recover sources.
xt = Ast
Source Mixture
-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4
s1
s2
x1
x2
Nonnegative independent component analysis
-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4
Step 1: Whiten
0
1
2
3
4
0 1 2 3 4-1-2-3-4-1
w1
w2
-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4
Step 2:Rotate
y1
y2
Source Mixture
Nonnegativesimilarity matching
s1
s2
x1
x2
Many%ways
Plumbley’s solution (2002)
Source Whitened mixture
Inner products (similarities) are preserved by orthogonal transformations.
Nonnegative similarity matching:
Pehlevan, Mohan & Chklovskii, 2017
0
1
2
3
4
0 1 2 3 4-1-2-3-4-1-1
0
1
2
3
4
0 1 2 3 4-1-2-3-4s1
s2
w1
w2
Finding optimal rotation is solving NSM
min8t,yt�0
TX
t=1
TX
t0=1
�wT
t wt0 � yTt yt0
�2
<latexit sha1_base64="DQggcDSDv/nNkHyVMo1u45wA3tE=">AAACZ3icbVFNa9wwEJXd5mvTZDdNKYVc1C4hCSTBDoW0h0Kglx5T2E0C610ja8e7IpLsSOMGY9wf2VvvveRfVLvr0OZjQPDmvZmR5inJpbAYBL89/8XLpeWV1bXW+quNzXZn6/WFzQrDoc8zmZmrhFmQQkMfBUq4yg0wlUi4TK6/zvTLH2CsyHQPyxyGik20SAVn6Ki48zNSQsdVlGaGSUnxkFZRktIyxjqawA0NahrZQsUVfgnrUe8+2WsyCSnuzztu6xhHvXvoKmp61Mz6p5QLJTJiMsWD0QmNO93gOJgHfQrCBnRJE+dx51c0znihQCOXzNpBGOQ4rJhBwSXUraiwkDN+zSYwcFAzBXZYzX2q6a5jxtSt6o5GOmf/76iYsrZUiatUDKf2sTYjn9MGBaafhpXQeYGg+eKitHB2ZnRmOh0LAxxl6QDjRri3Uj5lhnF0X9NyJoSPV34KLk6OQ4e/f+yefW7sWCU75APZJyE5JWfkGzknfcLJH6/lbXtvvDu/7b/13y1Kfa/p2SYPwn//F1CpuCU=</latexit><latexit sha1_base64="DQggcDSDv/nNkHyVMo1u45wA3tE=">AAACZ3icbVFNa9wwEJXd5mvTZDdNKYVc1C4hCSTBDoW0h0Kglx5T2E0C610ja8e7IpLsSOMGY9wf2VvvveRfVLvr0OZjQPDmvZmR5inJpbAYBL89/8XLpeWV1bXW+quNzXZn6/WFzQrDoc8zmZmrhFmQQkMfBUq4yg0wlUi4TK6/zvTLH2CsyHQPyxyGik20SAVn6Ki48zNSQsdVlGaGSUnxkFZRktIyxjqawA0NahrZQsUVfgnrUe8+2WsyCSnuzztu6xhHvXvoKmp61Mz6p5QLJTJiMsWD0QmNO93gOJgHfQrCBnRJE+dx51c0znihQCOXzNpBGOQ4rJhBwSXUraiwkDN+zSYwcFAzBXZYzX2q6a5jxtSt6o5GOmf/76iYsrZUiatUDKf2sTYjn9MGBaafhpXQeYGg+eKitHB2ZnRmOh0LAxxl6QDjRri3Uj5lhnF0X9NyJoSPV34KLk6OQ4e/f+yefW7sWCU75APZJyE5JWfkGzknfcLJH6/lbXtvvDu/7b/13y1Kfa/p2SYPwn//F1CpuCU=</latexit><latexit sha1_base64="DQggcDSDv/nNkHyVMo1u45wA3tE=">AAACZ3icbVFNa9wwEJXd5mvTZDdNKYVc1C4hCSTBDoW0h0Kglx5T2E0C610ja8e7IpLsSOMGY9wf2VvvveRfVLvr0OZjQPDmvZmR5inJpbAYBL89/8XLpeWV1bXW+quNzXZn6/WFzQrDoc8zmZmrhFmQQkMfBUq4yg0wlUi4TK6/zvTLH2CsyHQPyxyGik20SAVn6Ki48zNSQsdVlGaGSUnxkFZRktIyxjqawA0NahrZQsUVfgnrUe8+2WsyCSnuzztu6xhHvXvoKmp61Mz6p5QLJTJiMsWD0QmNO93gOJgHfQrCBnRJE+dx51c0znihQCOXzNpBGOQ4rJhBwSXUraiwkDN+zSYwcFAzBXZYzX2q6a5jxtSt6o5GOmf/76iYsrZUiatUDKf2sTYjn9MGBaafhpXQeYGg+eKitHB2ZnRmOh0LAxxl6QDjRri3Uj5lhnF0X9NyJoSPV34KLk6OQ4e/f+yefW7sWCU75APZJyE5JWfkGzknfcLJH6/lbXtvvDu/7b/13y1Kfa/p2SYPwn//F1CpuCU=</latexit><latexit sha1_base64="DQggcDSDv/nNkHyVMo1u45wA3tE=">AAACZ3icbVFNa9wwEJXd5mvTZDdNKYVc1C4hCSTBDoW0h0Kglx5T2E0C610ja8e7IpLsSOMGY9wf2VvvveRfVLvr0OZjQPDmvZmR5inJpbAYBL89/8XLpeWV1bXW+quNzXZn6/WFzQrDoc8zmZmrhFmQQkMfBUq4yg0wlUi4TK6/zvTLH2CsyHQPyxyGik20SAVn6Ki48zNSQsdVlGaGSUnxkFZRktIyxjqawA0NahrZQsUVfgnrUe8+2WsyCSnuzztu6xhHvXvoKmp61Mz6p5QLJTJiMsWD0QmNO93gOJgHfQrCBnRJE+dx51c0znihQCOXzNpBGOQ4rJhBwSXUraiwkDN+zSYwcFAzBXZYzX2q6a5jxtSt6o5GOmf/76iYsrZUiatUDKf2sTYjn9MGBaafhpXQeYGg+eKitHB2ZnRmOh0LAxxl6QDjRri3Uj5lhnF0X9NyJoSPV34KLk6OQ4e/f+yefW7sWCU75APZJyE5JWfkGzknfcLJH6/lbXtvvDu/7b/13y1Kfa/p2SYPwn//F1CpuCU=</latexit>
demonstration by Sreyas Mohan
decorrelationnonnegative
similarity matching
MixtureSource Recovered
Pehlevan & Chklovskii, NeurIPS, 2015Pehlevan, Mohan & Chklovskii, Neural Comp., 2017
!-clustering: Any dataset can be uniquely clustered such that a) nearest
neighbors within a cluster have ≥ ! similarity, and b) similarity of any two points
that belong to different clusters is < !.
Clustering
min8t,yt�0
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0 � ↵
�2
<latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit>
Theory result: In the limit of large-k, “α-clusters” are represented by disjoint sets of neurons.
Clustering
min8t,yt�0
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0 � ↵
�2
<latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit>
Theory result: In the limit of large-k, “α-clusters” are represented by disjoint sets of neurons.
��x
>t xt0 � ↵
�� y
>t yt0
�2<latexit sha1_base64="p6VyVxI7EA5Bp9PYrrxqBluxquA=">AAACRHicbVBLSwMxGMzWV62vqkcvwSLWQ8tuEdRbwYvHCvYB3bZk02wbmn2QfCsuS3+cF3+AN3+BFw+KeBXTdgXbOpAwmZmPJOOEgiswzRcjs7K6tr6R3cxtbe/s7uX3DxoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOqPrid+8Z1LxwL+DOGQdjwx87nJKQEu9fNsWzIXibE9sx8UP4x50bQjC31MCp+OSTUQ4JLbkgyGc4RKemvFcNJ5FcRrqVnr5glk2p8DLxEpJAaWo9fLPdj+gkcd8oIIo1bbMEDoJkcCpYOOcHSkWEjoiA9bW1CceU51kWsIYn2ilj91A6uUDnqp/JxLiKRV7jk56BIZq0ZuI/3ntCNzLTsL9MALm09lFbiQwBHjSKO5zySiIWBNCJddvxXRIJKGge8/pEqzFLy+TRqVsaX57XqhepXVk0RE6RkVkoQtURTeohuqIokf0it7Rh/FkvBmfxtcsmjHSmUM0B+P7B/uHsfY=</latexit><latexit sha1_base64="p6VyVxI7EA5Bp9PYrrxqBluxquA=">AAACRHicbVBLSwMxGMzWV62vqkcvwSLWQ8tuEdRbwYvHCvYB3bZk02wbmn2QfCsuS3+cF3+AN3+BFw+KeBXTdgXbOpAwmZmPJOOEgiswzRcjs7K6tr6R3cxtbe/s7uX3DxoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOqPrid+8Z1LxwL+DOGQdjwx87nJKQEu9fNsWzIXibE9sx8UP4x50bQjC31MCp+OSTUQ4JLbkgyGc4RKemvFcNJ5FcRrqVnr5glk2p8DLxEpJAaWo9fLPdj+gkcd8oIIo1bbMEDoJkcCpYOOcHSkWEjoiA9bW1CceU51kWsIYn2ilj91A6uUDnqp/JxLiKRV7jk56BIZq0ZuI/3ntCNzLTsL9MALm09lFbiQwBHjSKO5zySiIWBNCJddvxXRIJKGge8/pEqzFLy+TRqVsaX57XqhepXVk0RE6RkVkoQtURTeohuqIokf0it7Rh/FkvBmfxtcsmjHSmUM0B+P7B/uHsfY=</latexit><latexit sha1_base64="p6VyVxI7EA5Bp9PYrrxqBluxquA=">AAACRHicbVBLSwMxGMzWV62vqkcvwSLWQ8tuEdRbwYvHCvYB3bZk02wbmn2QfCsuS3+cF3+AN3+BFw+KeBXTdgXbOpAwmZmPJOOEgiswzRcjs7K6tr6R3cxtbe/s7uX3DxoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOqPrid+8Z1LxwL+DOGQdjwx87nJKQEu9fNsWzIXibE9sx8UP4x50bQjC31MCp+OSTUQ4JLbkgyGc4RKemvFcNJ5FcRrqVnr5glk2p8DLxEpJAaWo9fLPdj+gkcd8oIIo1bbMEDoJkcCpYOOcHSkWEjoiA9bW1CceU51kWsIYn2ilj91A6uUDnqp/JxLiKRV7jk56BIZq0ZuI/3ntCNzLTsL9MALm09lFbiQwBHjSKO5zySiIWBNCJddvxXRIJKGge8/pEqzFLy+TRqVsaX57XqhepXVk0RE6RkVkoQtURTeohuqIokf0it7Rh/FkvBmfxtcsmjHSmUM0B+P7B/uHsfY=</latexit><latexit sha1_base64="p6VyVxI7EA5Bp9PYrrxqBluxquA=">AAACRHicbVBLSwMxGMzWV62vqkcvwSLWQ8tuEdRbwYvHCvYB3bZk02wbmn2QfCsuS3+cF3+AN3+BFw+KeBXTdgXbOpAwmZmPJOOEgiswzRcjs7K6tr6R3cxtbe/s7uX3DxoqiCRldRqIQLYcopjgPqsDB8FaoWTEcwRrOqPrid+8Z1LxwL+DOGQdjwx87nJKQEu9fNsWzIXibE9sx8UP4x50bQjC31MCp+OSTUQ4JLbkgyGc4RKemvFcNJ5FcRrqVnr5glk2p8DLxEpJAaWo9fLPdj+gkcd8oIIo1bbMEDoJkcCpYOOcHSkWEjoiA9bW1CceU51kWsIYn2ilj91A6uUDnqp/JxLiKRV7jk56BIZq0ZuI/3ntCNzLTsL9MALm09lFbiQwBHjSKO5zySiIWBNCJddvxXRIJKGge8/pEqzFLy+TRqVsaX57XqhepXVk0RE6RkVkoQtURTeohuqIokf0it7Rh/FkvBmfxtcsmjHSmUM0B+P7B/uHsfY=</latexit>
2
6666664
145000
3
7777775
> 2
6666664
000374
3
7777775
<latexit sha1_base64="LzDM7IowstQrAirSSao4v1gqWiY=">AAACe3icdVHbSsQwEE3rfb2t+ihIcBFFRFovqG+CLz6u4KqwrUuane4G07QkU3Ep+xN+mm/+iS+C6W4RrwM5HM7MnEwmUSaFQc97ddyJyanpmdm52vzC4tJyfWX1xqS55tDiqUz1XcQMSKGghQIl3GUaWBJJuI0eLsr87SNoI1J1jYMMwoT1lIgFZ2ilTv05kBAjbdMggp5QRcJQi6ch9WkQ0KMSjkvwvgKo7mddoEWvj+F9gGlG//HyvjsclnBS+f/h1ak3vH1vFPQ38SvSIFU0O/WXoJvyPAGFXDJj2r6XYVgwjYJLGNaC3EDG+APrQdtSxRIwYTHa3ZBuWaVL41Tbo5CO1K8dBUuMGSSRrbRz9s3PXCn+lWvnGJ+GhVBZjqD4+KI4lxRTWn4E7QoNHOXAEsa1sLNS3meacbTfVbNL8H8++Te5Odj3Lb86apyfVeuYJetkk+wQn5yQc3JJmqRFOHlzNpxtZ8d5dxvurrs3LnWdqmeNfAv3+ANVmLtg</latexit><latexit sha1_base64="LzDM7IowstQrAirSSao4v1gqWiY=">AAACe3icdVHbSsQwEE3rfb2t+ihIcBFFRFovqG+CLz6u4KqwrUuane4G07QkU3Ep+xN+mm/+iS+C6W4RrwM5HM7MnEwmUSaFQc97ddyJyanpmdm52vzC4tJyfWX1xqS55tDiqUz1XcQMSKGghQIl3GUaWBJJuI0eLsr87SNoI1J1jYMMwoT1lIgFZ2ilTv05kBAjbdMggp5QRcJQi6ch9WkQ0KMSjkvwvgKo7mddoEWvj+F9gGlG//HyvjsclnBS+f/h1ak3vH1vFPQ38SvSIFU0O/WXoJvyPAGFXDJj2r6XYVgwjYJLGNaC3EDG+APrQdtSxRIwYTHa3ZBuWaVL41Tbo5CO1K8dBUuMGSSRrbRz9s3PXCn+lWvnGJ+GhVBZjqD4+KI4lxRTWn4E7QoNHOXAEsa1sLNS3meacbTfVbNL8H8++Te5Odj3Lb86apyfVeuYJetkk+wQn5yQc3JJmqRFOHlzNpxtZ8d5dxvurrs3LnWdqmeNfAv3+ANVmLtg</latexit><latexit sha1_base64="LzDM7IowstQrAirSSao4v1gqWiY=">AAACe3icdVHbSsQwEE3rfb2t+ihIcBFFRFovqG+CLz6u4KqwrUuane4G07QkU3Ep+xN+mm/+iS+C6W4RrwM5HM7MnEwmUSaFQc97ddyJyanpmdm52vzC4tJyfWX1xqS55tDiqUz1XcQMSKGghQIl3GUaWBJJuI0eLsr87SNoI1J1jYMMwoT1lIgFZ2ilTv05kBAjbdMggp5QRcJQi6ch9WkQ0KMSjkvwvgKo7mddoEWvj+F9gGlG//HyvjsclnBS+f/h1ak3vH1vFPQ38SvSIFU0O/WXoJvyPAGFXDJj2r6XYVgwjYJLGNaC3EDG+APrQdtSxRIwYTHa3ZBuWaVL41Tbo5CO1K8dBUuMGSSRrbRz9s3PXCn+lWvnGJ+GhVBZjqD4+KI4lxRTWn4E7QoNHOXAEsa1sLNS3meacbTfVbNL8H8++Te5Odj3Lb86apyfVeuYJetkk+wQn5yQc3JJmqRFOHlzNpxtZ8d5dxvurrs3LnWdqmeNfAv3+ANVmLtg</latexit><latexit sha1_base64="LzDM7IowstQrAirSSao4v1gqWiY=">AAACe3icdVHbSsQwEE3rfb2t+ihIcBFFRFovqG+CLz6u4KqwrUuane4G07QkU3Ep+xN+mm/+iS+C6W4RrwM5HM7MnEwmUSaFQc97ddyJyanpmdm52vzC4tJyfWX1xqS55tDiqUz1XcQMSKGghQIl3GUaWBJJuI0eLsr87SNoI1J1jYMMwoT1lIgFZ2ilTv05kBAjbdMggp5QRcJQi6ch9WkQ0KMSjkvwvgKo7mddoEWvj+F9gGlG//HyvjsclnBS+f/h1ak3vH1vFPQ38SvSIFU0O/WXoJvyPAGFXDJj2r6XYVgwjYJLGNaC3EDG+APrQdtSxRIwYTHa3ZBuWaVL41Tbo5CO1K8dBUuMGSSRrbRz9s3PXCn+lWvnGJ+GhVBZjqD4+KI4lxRTWn4E7QoNHOXAEsa1sLNS3meacbTfVbNL8H8++Te5Odj3Lb86apyfVeuYJetkk+wQn5yQc3JJmqRFOHlzNpxtZ8d5dxvurrs3LnWdqmeNfAv3+ANVmLtg</latexit>
Clustering - example
input
x1
xn. . .
y1
y2x2
yk
. . .
. . .
output response
neuro
n #
(y i)
Theory result: In the continuum limit, a circular data set is tiled by localized bumps.
input
Resp
onse
min8t,yt�0
TX
t=1
TX
t0=1
�x
>t xt0 � y
>t yt0 � ↵
�2
<latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit><latexit sha1_base64="NKbs1v0GHHV2HtgUz5w1osu7C7Y=">AAACdXicbVFNa9wwEJXdj6SbtN20l0IJqNl+pJAPOwTaHAqBXHpMIZsEVrtG1o53RSTZlcYlxvgf9Nf11r+RS67R7jrQJB0QvHlvRtK8SQslHUbR3yB89PjJ06XlZ52V1ecvXnbXXp26vLQC+iJXuT1PuQMlDfRRooLzwgLXqYKz9OJopp/9Autkbk6wKmCo+cTITAqOnkq6v5mWJqlZlluuFMUtWrM0o1WCDZvATxo1lLlSJzV+i5vRyW3yqc0UZLg577hsEhwxzIvbzBc1dLu97o5YLcRtxlUx5ZRZOZni59EeTbq9aCeaB30I4hb0SBvHSfcPG+ei1GBQKO7cII4KHNbcohQKmg4rHRRcXPAJDDw0XIMb1nPXGvrBM2PqB/fHIJ2z/3bUXDtX6dRXao5Td1+bkf/TBiVmX4e1NEWJYMTioaz05uZ0tgI6lhYEqsoDLqz0f6Viyi0X6BfV8SbE90d+CE73dmKPf+z3Dg9aO5bJW7JBNklMvpBD8p0ckz4R5Cp4E7wLNoLrcD18H35clIZB2/Oa3Ilw9wZN5L4Y</latexit>
�⇡<latexit sha1_base64="9yme3tnzhjNqXO6v45lobNoAz/M=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDuCm5cVrAXaIeSSTNtaJIJSUYoQ1/BjQtF3PpC7nwbM+0stPWHwMd/ziHn/JHizFjf//ZKa+sbm1vl7crO7t7+QfXwqG2SVBPaIglPdDfChnImacsyy2lXaYpFxGknmtzl9c4T1YYl8tFOFQ0FHkkWM4Jtbl30FRtUa37dnwutQlBADQo1B9Wv/jAhqaDSEo6N6QW+smGGtWWE01mlnxqqMJngEe05lFhQE2bzXWfozDlDFCfaPWnR3P09kWFhzFRErlNgOzbLtdz8r9ZLbXwTZkyq1FJJFh/FKUc2QfnhaMg0JZZPHWCimdsVkTHWmFgXT8WFECyfvArty3rg+OGq1rgt4ijDCZzCOQRwDQ24hya0gMAYnuEV3jzhvXjv3seiteQVM8fwR97nD7atjfs=</latexit><latexit sha1_base64="9yme3tnzhjNqXO6v45lobNoAz/M=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDuCm5cVrAXaIeSSTNtaJIJSUYoQ1/BjQtF3PpC7nwbM+0stPWHwMd/ziHn/JHizFjf//ZKa+sbm1vl7crO7t7+QfXwqG2SVBPaIglPdDfChnImacsyy2lXaYpFxGknmtzl9c4T1YYl8tFOFQ0FHkkWM4Jtbl30FRtUa37dnwutQlBADQo1B9Wv/jAhqaDSEo6N6QW+smGGtWWE01mlnxqqMJngEe05lFhQE2bzXWfozDlDFCfaPWnR3P09kWFhzFRErlNgOzbLtdz8r9ZLbXwTZkyq1FJJFh/FKUc2QfnhaMg0JZZPHWCimdsVkTHWmFgXT8WFECyfvArty3rg+OGq1rgt4ijDCZzCOQRwDQ24hya0gMAYnuEV3jzhvXjv3seiteQVM8fwR97nD7atjfs=</latexit><latexit sha1_base64="9yme3tnzhjNqXO6v45lobNoAz/M=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDuCm5cVrAXaIeSSTNtaJIJSUYoQ1/BjQtF3PpC7nwbM+0stPWHwMd/ziHn/JHizFjf//ZKa+sbm1vl7crO7t7+QfXwqG2SVBPaIglPdDfChnImacsyy2lXaYpFxGknmtzl9c4T1YYl8tFOFQ0FHkkWM4Jtbl30FRtUa37dnwutQlBADQo1B9Wv/jAhqaDSEo6N6QW+smGGtWWE01mlnxqqMJngEe05lFhQE2bzXWfozDlDFCfaPWnR3P09kWFhzFRErlNgOzbLtdz8r9ZLbXwTZkyq1FJJFh/FKUc2QfnhaMg0JZZPHWCimdsVkTHWmFgXT8WFECyfvArty3rg+OGq1rgt4ijDCZzCOQRwDQ24hya0gMAYnuEV3jzhvXjv3seiteQVM8fwR97nD7atjfs=</latexit><latexit sha1_base64="9yme3tnzhjNqXO6v45lobNoAz/M=">AAAB63icbZDLSgMxFIbP1Futt6pLN8EiuLHMiKDuCm5cVrAXaIeSSTNtaJIJSUYoQ1/BjQtF3PpC7nwbM+0stPWHwMd/ziHn/JHizFjf//ZKa+sbm1vl7crO7t7+QfXwqG2SVBPaIglPdDfChnImacsyy2lXaYpFxGknmtzl9c4T1YYl8tFOFQ0FHkkWM4Jtbl30FRtUa37dnwutQlBADQo1B9Wv/jAhqaDSEo6N6QW+smGGtWWE01mlnxqqMJngEe05lFhQE2bzXWfozDlDFCfaPWnR3P09kWFhzFRErlNgOzbLtdz8r9ZLbXwTZkyq1FJJFh/FKUc2QfnhaMg0JZZPHWCimdsVkTHWmFgXT8WFECyfvArty3rg+OGq1rgt4ijDCZzCOQRwDQ24hya0gMAYnuEV3jzhvXjv3seiteQVM8fwR97nD7atjfs=</latexit> ⇡
<latexit sha1_base64="w6OPujM7UTNOnq4WK6+3jCQJLwA=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRbwYvHivYD2lA220m7dLMJuxuhhP4ELx4U8eov8ua/cZvmoK0vLDy8M8POvEEiuDau++2U1tY3NrfK25Wd3b39g+rhUVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk9t5vfOESvNYPpppgn5ER5KHnFFjrYd+wgfVmlt3c5FV8AqoQaHmoPrVH8YsjVAaJqjWPc9NjJ9RZTgTOKv0U40JZRM6wp5FSSPUfpavOiNn1hmSMFb2SUNy9/dERiOtp1FgOyNqxnq5Njf/q/VSE177GZdJalCyxUdhKoiJyfxuMuQKmRFTC5QpbnclbEwVZcamU7EheMsnr0L7ou5Zvr+sNW6KOMpwAqdwDh5cQQPuoAktYDCCZ3iFN0c4L86787FoLTnFzDH8kfP5A00KjcQ=</latexit><latexit sha1_base64="w6OPujM7UTNOnq4WK6+3jCQJLwA=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRbwYvHivYD2lA220m7dLMJuxuhhP4ELx4U8eov8ua/cZvmoK0vLDy8M8POvEEiuDau++2U1tY3NrfK25Wd3b39g+rhUVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk9t5vfOESvNYPpppgn5ER5KHnFFjrYd+wgfVmlt3c5FV8AqoQaHmoPrVH8YsjVAaJqjWPc9NjJ9RZTgTOKv0U40JZRM6wp5FSSPUfpavOiNn1hmSMFb2SUNy9/dERiOtp1FgOyNqxnq5Njf/q/VSE177GZdJalCyxUdhKoiJyfxuMuQKmRFTC5QpbnclbEwVZcamU7EheMsnr0L7ou5Zvr+sNW6KOMpwAqdwDh5cQQPuoAktYDCCZ3iFN0c4L86787FoLTnFzDH8kfP5A00KjcQ=</latexit><latexit sha1_base64="w6OPujM7UTNOnq4WK6+3jCQJLwA=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRbwYvHivYD2lA220m7dLMJuxuhhP4ELx4U8eov8ua/cZvmoK0vLDy8M8POvEEiuDau++2U1tY3NrfK25Wd3b39g+rhUVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk9t5vfOESvNYPpppgn5ER5KHnFFjrYd+wgfVmlt3c5FV8AqoQaHmoPrVH8YsjVAaJqjWPc9NjJ9RZTgTOKv0U40JZRM6wp5FSSPUfpavOiNn1hmSMFb2SUNy9/dERiOtp1FgOyNqxnq5Njf/q/VSE177GZdJalCyxUdhKoiJyfxuMuQKmRFTC5QpbnclbEwVZcamU7EheMsnr0L7ou5Zvr+sNW6KOMpwAqdwDh5cQQPuoAktYDCCZ3iFN0c4L86787FoLTnFzDH8kfP5A00KjcQ=</latexit><latexit sha1_base64="w6OPujM7UTNOnq4WK6+3jCQJLwA=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEdRbwYvHivYD2lA220m7dLMJuxuhhP4ELx4U8eov8ua/cZvmoK0vLDy8M8POvEEiuDau++2U1tY3NrfK25Wd3b39g+rhUVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk9t5vfOESvNYPpppgn5ER5KHnFFjrYd+wgfVmlt3c5FV8AqoQaHmoPrVH8YsjVAaJqjWPc9NjJ9RZTgTOKv0U40JZRM6wp5FSSPUfpavOiNn1hmSMFb2SUNy9/dERiOtp1FgOyNqxnq5Njf/q/VSE177GZdJalCyxUdhKoiJyfxuMuQKmRFTC5QpbnclbEwVZcamU7EheMsnr0L7ou5Zvr+sNW6KOMpwAqdwDh5cQQPuoAktYDCCZ3iFN0c4L86787FoLTnFzDH8kfP5A00KjcQ=</latexit> ✓<latexit sha1_base64="aO29ME9arMnSlsyKaAHeo4RUYXc=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RVBvBS8eK9gPaJeSTdM2NpssyaxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5RIYdH3v73C2vrG5lZxu7Szu7d/UD48alqdGsYbTEtt2hG1XArFGyhQ8nZiOI0jyVvR+HZWbz1xY4VWDzhJeBjToRIDwSg6q9nFEUfaK1f8qj8XWYUghwrkqvfKX92+ZmnMFTJJre0EfoJhRg0KJvm01E0tTygb0yHvOFQ05jbM5ttOyZlz+mSgjXsKydz9PZHR2NpJHLnOmOLILtdm5n+1ToqD6zATKkmRK7b4aJBKgprMTid9YThDOXFAmRFuV8JG1FCGLqCSCyFYPnkVmhfVwPH9ZaV2k8dRhBM4hXMI4ApqcAd1aACDR3iGV3jztPfivXsfi9aCl88cwx95nz+hRI8f</latexit><latexit sha1_base64="aO29ME9arMnSlsyKaAHeo4RUYXc=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RVBvBS8eK9gPaJeSTdM2NpssyaxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5RIYdH3v73C2vrG5lZxu7Szu7d/UD48alqdGsYbTEtt2hG1XArFGyhQ8nZiOI0jyVvR+HZWbz1xY4VWDzhJeBjToRIDwSg6q9nFEUfaK1f8qj8XWYUghwrkqvfKX92+ZmnMFTJJre0EfoJhRg0KJvm01E0tTygb0yHvOFQ05jbM5ttOyZlz+mSgjXsKydz9PZHR2NpJHLnOmOLILtdm5n+1ToqD6zATKkmRK7b4aJBKgprMTid9YThDOXFAmRFuV8JG1FCGLqCSCyFYPnkVmhfVwPH9ZaV2k8dRhBM4hXMI4ApqcAd1aACDR3iGV3jztPfivXsfi9aCl88cwx95nz+hRI8f</latexit><latexit sha1_base64="aO29ME9arMnSlsyKaAHeo4RUYXc=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RVBvBS8eK9gPaJeSTdM2NpssyaxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5RIYdH3v73C2vrG5lZxu7Szu7d/UD48alqdGsYbTEtt2hG1XArFGyhQ8nZiOI0jyVvR+HZWbz1xY4VWDzhJeBjToRIDwSg6q9nFEUfaK1f8qj8XWYUghwrkqvfKX92+ZmnMFTJJre0EfoJhRg0KJvm01E0tTygb0yHvOFQ05jbM5ttOyZlz+mSgjXsKydz9PZHR2NpJHLnOmOLILtdm5n+1ToqD6zATKkmRK7b4aJBKgprMTid9YThDOXFAmRFuV8JG1FCGLqCSCyFYPnkVmhfVwPH9ZaV2k8dRhBM4hXMI4ApqcAd1aACDR3iGV3jztPfivXsfi9aCl88cwx95nz+hRI8f</latexit><latexit sha1_base64="aO29ME9arMnSlsyKaAHeo4RUYXc=">AAAB7XicbZBNSwMxEIZn61etX1WPXoJF8FR2RVBvBS8eK9gPaJeSTdM2NpssyaxQlv4HLx4U8er/8ea/MW33oK0vBB7emSEzb5RIYdH3v73C2vrG5lZxu7Szu7d/UD48alqdGsYbTEtt2hG1XArFGyhQ8nZiOI0jyVvR+HZWbz1xY4VWDzhJeBjToRIDwSg6q9nFEUfaK1f8qj8XWYUghwrkqvfKX92+ZmnMFTJJre0EfoJhRg0KJvm01E0tTygb0yHvOFQ05jbM5ttOyZlz+mSgjXsKydz9PZHR2NpJHLnOmOLILtdm5n+1ToqD6zATKkmRK7b4aJBKgprMTid9YThDOXFAmRFuV8JG1FCGLqCSCyFYPnkVmhfVwPH9ZaV2k8dRhBM4hXMI4ApqcAd1aACDR3iGV3jztPfivXsfi9aCl88cwx95nz+hRI8f</latexit>
Sengupta, Tepper, Pehlevan, Genkin, Chkvlovskii, NeurIPS 2018
Manifold learning by tiling
Manifold learning by tiling
x1
xn
. . .
y1
y2x2
yk
. . .
. . .
0 30 60 90 120 150 180 210 240 270 300 330
20
40
60
80
100
120
140
160
180
200
neuro
n #
(y i)
input output response
angle
Manifold learning by tiling – more examples
Sengupta, Tepper, Pehlevan, Genkin, Chkvlovskii, NeurIPS 2018
MNIST
2-sphere
“Respecting biological constraints, viewed naively as a handicap in developing
competitive general-purpose machine learning algorithms, can instead
facilitate the development of artificial NNs by restricting the search space of
possible algorithms.”
From a blog post with Mitya Chklovskii to appear in www.offconvex.org
Dmitri ChklovskiiAnirvan Sengupta
Alex GenkinMariano Tepper
Andrea GiovannucciVictor MindenSreyas Mohan
Nikolai ChapochnikovXinyuan Zhao
Tao Hu
Acknowledgments
THANK YOU!