linear regression and k-nearest neighborsbryce/cs63/s18/slides/3-28_knn.pdf · linear regression...
TRANSCRIPT
LinearRegressionandK-NearestNeighbors
3/28/18
LinearRegressionHypothesisSpace
Supervisedlearning• Foreveryinputinthedataset,weknowtheoutput
Regression• Outputsarecontinuous• Anumber,notacategorylabel
Thelearnedmodel:• Alinearfunctionmappinginputtooutput• Aweightforeachfeature(includingbias)
LinearRegression
Wewanttofindthelinearmodelthatfitsourdatabest.
Keyidea:modeldataaslinearfunctionplusnoise.Picktheweightstominimizenoisemagnitude.
f(~x) =
2
6664
wb
w0...wd
3
7775·
2
6664
1x0...xd
3
7775+ ✏
SquaredError
f̂(~x) =
2
6664
wb
w0...wd
3
7775·
2
6664
1x0...xd
3
7775f(~x) =
2
6664
wb
w0...wd
3
7775·
2
6664
1x0...xd
3
7775+ ✏
Defineerrorforadatapointtobethesquareddistancebetweencorrectoutputandpredictedoutput:
Errorforthemodelisthesumofpointerrors:
⇣f(~x)� f̂(~x)
⌘2= ✏2
X
~x2data
⇣y � f̂(~x)
⌘=
X
~x2data
✏2~x
Errorasafunction
X :inputexamplesY :outputexamples
:learnedweights:modelprediction
Errordependsonthedataandtheweights.
Foragivendataset,errorisafunctionoftheweights.
X ⌘�~x
Y ⌘�f(x) 8~x 2 X
<latexit sha1_base64="ufLf4iJY4ipiU3WmXuJfVY+wyaQ=">AAACiHicbVHRbtMwFHUyGCMDVsYjL1dUQ500qgQBYxpIE7zwOCTKguqqctybzppjB9vpVkX5F76JN/4GNwuIdVzJ8vG55/heX2elFNbF8a8g3Lhzd/Pe1v1o+8HDRzu9x7tfra4MxxHXUps0YxalUDhywklMS4OsyCSeZRcfV/mzBRortPriliVOCjZXIhecOU9Nez/2KJZWSK0G6QHQBfL6stmH90BtVUzrlrhqgAoFaQP54GofXvyRAeUz7aDTRCk8p/i9EgugmZjTGv6aV0e/0ejbmqS9jx7TXBsmJdyo1tmmvX48jNuA2yDpQJ90cTrt/aQzzasCleOSWTtO4tJNamac4BKbiFYWS8Yv2BzHHipWoJ3U7SQb2PPMDHw7fikHLfuvo2aFtcsi88qCuXO7nluR/8uNK5e/ndRClZVDxa8L5ZUEp2H1LTATBrmTSw8YN8L3CvycGcad/7zIDyFZf/JtMHo5PBomn1/1T95109giT8kzMiAJOSQn5BM5JSPCg83gIHgdvAm3wyQ8DI+upWHQeZ6QGxF++A1hcsHY</latexit><latexit sha1_base64="ufLf4iJY4ipiU3WmXuJfVY+wyaQ=">AAACiHicbVHRbtMwFHUyGCMDVsYjL1dUQ500qgQBYxpIE7zwOCTKguqqctybzppjB9vpVkX5F76JN/4GNwuIdVzJ8vG55/heX2elFNbF8a8g3Lhzd/Pe1v1o+8HDRzu9x7tfra4MxxHXUps0YxalUDhywklMS4OsyCSeZRcfV/mzBRortPriliVOCjZXIhecOU9Nez/2KJZWSK0G6QHQBfL6stmH90BtVUzrlrhqgAoFaQP54GofXvyRAeUz7aDTRCk8p/i9EgugmZjTGv6aV0e/0ejbmqS9jx7TXBsmJdyo1tmmvX48jNuA2yDpQJ90cTrt/aQzzasCleOSWTtO4tJNamac4BKbiFYWS8Yv2BzHHipWoJ3U7SQb2PPMDHw7fikHLfuvo2aFtcsi88qCuXO7nluR/8uNK5e/ndRClZVDxa8L5ZUEp2H1LTATBrmTSw8YN8L3CvycGcad/7zIDyFZf/JtMHo5PBomn1/1T95109giT8kzMiAJOSQn5BM5JSPCg83gIHgdvAm3wyQ8DI+upWHQeZ6QGxF++A1hcsHY</latexit><latexit sha1_base64="ufLf4iJY4ipiU3WmXuJfVY+wyaQ=">AAACiHicbVHRbtMwFHUyGCMDVsYjL1dUQ500qgQBYxpIE7zwOCTKguqqctybzppjB9vpVkX5F76JN/4GNwuIdVzJ8vG55/heX2elFNbF8a8g3Lhzd/Pe1v1o+8HDRzu9x7tfra4MxxHXUps0YxalUDhywklMS4OsyCSeZRcfV/mzBRortPriliVOCjZXIhecOU9Nez/2KJZWSK0G6QHQBfL6stmH90BtVUzrlrhqgAoFaQP54GofXvyRAeUz7aDTRCk8p/i9EgugmZjTGv6aV0e/0ejbmqS9jx7TXBsmJdyo1tmmvX48jNuA2yDpQJ90cTrt/aQzzasCleOSWTtO4tJNamac4BKbiFYWS8Yv2BzHHipWoJ3U7SQb2PPMDHw7fikHLfuvo2aFtcsi88qCuXO7nluR/8uNK5e/ndRClZVDxa8L5ZUEp2H1LTATBrmTSw8YN8L3CvycGcad/7zIDyFZf/JtMHo5PBomn1/1T95109giT8kzMiAJOSQn5BM5JSPCg83gIHgdvAm3wyQ8DI+upWHQeZ6QGxF++A1hcsHY</latexit><latexit sha1_base64="ufLf4iJY4ipiU3WmXuJfVY+wyaQ=">AAACiHicbVHRbtMwFHUyGCMDVsYjL1dUQ500qgQBYxpIE7zwOCTKguqqctybzppjB9vpVkX5F76JN/4GNwuIdVzJ8vG55/heX2elFNbF8a8g3Lhzd/Pe1v1o+8HDRzu9x7tfra4MxxHXUps0YxalUDhywklMS4OsyCSeZRcfV/mzBRortPriliVOCjZXIhecOU9Nez/2KJZWSK0G6QHQBfL6stmH90BtVUzrlrhqgAoFaQP54GofXvyRAeUz7aDTRCk8p/i9EgugmZjTGv6aV0e/0ejbmqS9jx7TXBsmJdyo1tmmvX48jNuA2yDpQJ90cTrt/aQzzasCleOSWTtO4tJNamac4BKbiFYWS8Yv2BzHHipWoJ3U7SQb2PPMDHw7fikHLfuvo2aFtcsi88qCuXO7nluR/8uNK5e/ndRClZVDxa8L5ZUEp2H1LTATBrmTSw8YN8L3CvycGcad/7zIDyFZf/JtMHo5PBomn1/1T95109giT8kzMiAJOSQn5BM5JSPCg83gIHgdvAm3wyQ8DI+upWHQeZ6QGxF++A1hcsHY</latexit>
~w<latexit sha1_base64="AJftwNFD3TvriOY73TaUfUOxV4c=">AAACknicbVFdb9MwFHUyPkb4ykA88XJFVdRJUCUIbUwDUbQXHngYEmVBdVU57k1nzbGz2Cmrovwhfg5v/BvcLCDWcSXLx+ee43t9nRZSGBtFvzx/68bNW7e37wR3791/8DDcefTV6KrkOOZa6jJJmUEpFI6tsBKTokSWpxJP0rOjdf5kiaURWn2xqwKnOVsokQnOrKNm4Q+6RF5/b4I+xcIIqdUgeQEduQvvgJoqn9UtcdEAFQqSBrLBxS68/CMDyufaQqcJ+gk8p3heiSXQVCxoDX/d66PbaND/tqFpb6SHNNMlkxKu1Ot8s7AXDaM24DqIO9AjXRzPwp90rnmVo7JcMmMmcVTYac1KK7jEJqCVwYLxM7bAiYOK5WimdTvTBvqOmYNrxy1loWX/ddQsN2aVp06ZM3tqNnNr8n+5SWWzN9NaqKKyqPhloaySYDWsPwjmokRu5coBxkvhegV+ykrGrfvGwA0h3nzydTB+NTwYxp9f90Zvu2lsk6fkGRmQmOyTEflIjsmYcC/09rz33sh/4h/6H/yjS6nvdZ7H5Er4n34DSCvFmQ==</latexit><latexit sha1_base64="AJftwNFD3TvriOY73TaUfUOxV4c=">AAACknicbVFdb9MwFHUyPkb4ykA88XJFVdRJUCUIbUwDUbQXHngYEmVBdVU57k1nzbGz2Cmrovwhfg5v/BvcLCDWcSXLx+ee43t9nRZSGBtFvzx/68bNW7e37wR3791/8DDcefTV6KrkOOZa6jJJmUEpFI6tsBKTokSWpxJP0rOjdf5kiaURWn2xqwKnOVsokQnOrKNm4Q+6RF5/b4I+xcIIqdUgeQEduQvvgJoqn9UtcdEAFQqSBrLBxS68/CMDyufaQqcJ+gk8p3heiSXQVCxoDX/d66PbaND/tqFpb6SHNNMlkxKu1Ot8s7AXDaM24DqIO9AjXRzPwp90rnmVo7JcMmMmcVTYac1KK7jEJqCVwYLxM7bAiYOK5WimdTvTBvqOmYNrxy1loWX/ddQsN2aVp06ZM3tqNnNr8n+5SWWzN9NaqKKyqPhloaySYDWsPwjmokRu5coBxkvhegV+ykrGrfvGwA0h3nzydTB+NTwYxp9f90Zvu2lsk6fkGRmQmOyTEflIjsmYcC/09rz33sh/4h/6H/yjS6nvdZ7H5Er4n34DSCvFmQ==</latexit><latexit sha1_base64="AJftwNFD3TvriOY73TaUfUOxV4c=">AAACknicbVFdb9MwFHUyPkb4ykA88XJFVdRJUCUIbUwDUbQXHngYEmVBdVU57k1nzbGz2Cmrovwhfg5v/BvcLCDWcSXLx+ee43t9nRZSGBtFvzx/68bNW7e37wR3791/8DDcefTV6KrkOOZa6jJJmUEpFI6tsBKTokSWpxJP0rOjdf5kiaURWn2xqwKnOVsokQnOrKNm4Q+6RF5/b4I+xcIIqdUgeQEduQvvgJoqn9UtcdEAFQqSBrLBxS68/CMDyufaQqcJ+gk8p3heiSXQVCxoDX/d66PbaND/tqFpb6SHNNMlkxKu1Ot8s7AXDaM24DqIO9AjXRzPwp90rnmVo7JcMmMmcVTYac1KK7jEJqCVwYLxM7bAiYOK5WimdTvTBvqOmYNrxy1loWX/ddQsN2aVp06ZM3tqNnNr8n+5SWWzN9NaqKKyqPhloaySYDWsPwjmokRu5coBxkvhegV+ykrGrfvGwA0h3nzydTB+NTwYxp9f90Zvu2lsk6fkGRmQmOyTEflIjsmYcC/09rz33sh/4h/6H/yjS6nvdZ7H5Er4n34DSCvFmQ==</latexit><latexit sha1_base64="AJftwNFD3TvriOY73TaUfUOxV4c=">AAACknicbVFdb9MwFHUyPkb4ykA88XJFVdRJUCUIbUwDUbQXHngYEmVBdVU57k1nzbGz2Cmrovwhfg5v/BvcLCDWcSXLx+ee43t9nRZSGBtFvzx/68bNW7e37wR3791/8DDcefTV6KrkOOZa6jJJmUEpFI6tsBKTokSWpxJP0rOjdf5kiaURWn2xqwKnOVsokQnOrKNm4Q+6RF5/b4I+xcIIqdUgeQEduQvvgJoqn9UtcdEAFQqSBrLBxS68/CMDyufaQqcJ+gk8p3heiSXQVCxoDX/d66PbaND/tqFpb6SHNNMlkxKu1Ot8s7AXDaM24DqIO9AjXRzPwp90rnmVo7JcMmMmcVTYac1KK7jEJqCVwYLxM7bAiYOK5WimdTvTBvqOmYNrxy1loWX/ddQsN2aVp06ZM3tqNnNr8n+5SWWzN9NaqKKyqPhloaySYDWsPwjmokRu5coBxkvhegV+ykrGrfvGwA0h3nzydTB+NTwYxp9f90Zvu2lsk6fkGRmQmOyTEflIjsmYcC/09rz33sh/4h/6H/yjS6nvdZ7H5Er4n34DSCvFmQ==</latexit>
f̂(~x) ⌘ ~w · ~x<latexit sha1_base64="4kSqaTiDrqcSb6HSX7CQUmz9M9c=">AAACwXicdVFbb9MwFHbCZSPcCjzyckRV1EpQJQhpIEBMggceh0RZUF1VjnvSmjl2sJ2yKsqf5G3/BjfLEOvgSJa/853vXOyTlVJYF8dnQXjt+o2be/u3ott37t6733vw8KvVleE44Vpqk2bMohQKJ044iWlpkBWZxOPs5MM2frxGY4VWX9ymxFnBlkrkgjPnqXnvjK6Yq/NmSNfI69NmBBR/VGINrf+zAcoX2kEXjQYd7QGWVkithumzC+0I3gG1VTGvOzlQoSBtIB+ejuD5/0qm8PSiZyaWtIY/2VvXXzQafNvRtBXpG5prw6SES/26vHmvH4/j1uAqSDrQJ50dzXu/6ELzqkDluGTWTpO4dLOaGSe4xCailcWS8RO2xKmHihVoZ3W7ggYGnlmAH8cf5aBl/86oWWHtpsi8smBuZXdjW/JfsWnl8lezWqiycqj4eaO8kuA0bPcJC2GQO7nxgHEj/KzAV8ww7vzWI/8Jye6Tr4LJi/HrcfL5Zf/wbfcb++QxeUKGJCEH5JB8IkdkQnjwPsBABTr8GH4Py9CcS8Ogy3lELllY/wZWD9kP</latexit><latexit sha1_base64="4kSqaTiDrqcSb6HSX7CQUmz9M9c=">AAACwXicdVFbb9MwFHbCZSPcCjzyckRV1EpQJQhpIEBMggceh0RZUF1VjnvSmjl2sJ2yKsqf5G3/BjfLEOvgSJa/853vXOyTlVJYF8dnQXjt+o2be/u3ott37t6733vw8KvVleE44Vpqk2bMohQKJ044iWlpkBWZxOPs5MM2frxGY4VWX9ymxFnBlkrkgjPnqXnvjK6Yq/NmSNfI69NmBBR/VGINrf+zAcoX2kEXjQYd7QGWVkithumzC+0I3gG1VTGvOzlQoSBtIB+ejuD5/0qm8PSiZyaWtIY/2VvXXzQafNvRtBXpG5prw6SES/26vHmvH4/j1uAqSDrQJ50dzXu/6ELzqkDluGTWTpO4dLOaGSe4xCailcWS8RO2xKmHihVoZ3W7ggYGnlmAH8cf5aBl/86oWWHtpsi8smBuZXdjW/JfsWnl8lezWqiycqj4eaO8kuA0bPcJC2GQO7nxgHEj/KzAV8ww7vzWI/8Jye6Tr4LJi/HrcfL5Zf/wbfcb++QxeUKGJCEH5JB8IkdkQnjwPsBABTr8GH4Py9CcS8Ogy3lELllY/wZWD9kP</latexit><latexit sha1_base64="4kSqaTiDrqcSb6HSX7CQUmz9M9c=">AAACwXicdVFbb9MwFHbCZSPcCjzyckRV1EpQJQhpIEBMggceh0RZUF1VjnvSmjl2sJ2yKsqf5G3/BjfLEOvgSJa/853vXOyTlVJYF8dnQXjt+o2be/u3ott37t6733vw8KvVleE44Vpqk2bMohQKJ044iWlpkBWZxOPs5MM2frxGY4VWX9ymxFnBlkrkgjPnqXnvjK6Yq/NmSNfI69NmBBR/VGINrf+zAcoX2kEXjQYd7QGWVkithumzC+0I3gG1VTGvOzlQoSBtIB+ejuD5/0qm8PSiZyaWtIY/2VvXXzQafNvRtBXpG5prw6SES/26vHmvH4/j1uAqSDrQJ50dzXu/6ELzqkDluGTWTpO4dLOaGSe4xCailcWS8RO2xKmHihVoZ3W7ggYGnlmAH8cf5aBl/86oWWHtpsi8smBuZXdjW/JfsWnl8lezWqiycqj4eaO8kuA0bPcJC2GQO7nxgHEj/KzAV8ww7vzWI/8Jye6Tr4LJi/HrcfL5Zf/wbfcb++QxeUKGJCEH5JB8IkdkQnjwPsBABTr8GH4Py9CcS8Ogy3lELllY/wZWD9kP</latexit><latexit sha1_base64="4kSqaTiDrqcSb6HSX7CQUmz9M9c=">AAACwXicdVFbb9MwFHbCZSPcCjzyckRV1EpQJQhpIEBMggceh0RZUF1VjnvSmjl2sJ2yKsqf5G3/BjfLEOvgSJa/853vXOyTlVJYF8dnQXjt+o2be/u3ott37t6733vw8KvVleE44Vpqk2bMohQKJ044iWlpkBWZxOPs5MM2frxGY4VWX9ymxFnBlkrkgjPnqXnvjK6Yq/NmSNfI69NmBBR/VGINrf+zAcoX2kEXjQYd7QGWVkithumzC+0I3gG1VTGvOzlQoSBtIB+ejuD5/0qm8PSiZyaWtIY/2VvXXzQafNvRtBXpG5prw6SES/26vHmvH4/j1uAqSDrQJ50dzXu/6ELzqkDluGTWTpO4dLOaGSe4xCailcWS8RO2xKmHihVoZ3W7ggYGnlmAH8cf5aBl/86oWWHtpsi8smBuZXdjW/JfsWnl8lezWqiycqj4eaO8kuA0bPcJC2GQO7nxgHEj/KzAV8ww7vzWI/8Jye6Tr4LJi/HrcfL5Zf/wbfcb++QxeUKGJCEH5JB8IkdkQnjwPsBABTr8GH4Py9CcS8Ogy3lELllY/wZWD9kP</latexit>
f̂(~x)<latexit sha1_base64="MbprTF7I+1x4HKW5VW+Vg1SCS6Q=">AAAC03icdVJbb9MwFHbCbYRbgUdejqiKWgmqBCENBEiTeOFxk+gWVFeV45601hw7i51uVYgEiFf+HG/8Bv4Ebpoh1sGRLH/nO9+5+JLkUhgbhj89/8rVa9dv7NwMbt2+c/de5/6DQ6PLguOIa6mLOGEGpVA4ssJKjPMCWZZIPEqO363jR0ssjNDqg13lOMnYXIlUcGYdNe38ogtmq7Tu0yXy6qweBL1tBiielGIJjX9aA+UzbaGNOvmGdgBzI6RW/fjpuXYAb4GaMptWrRyoUBDXkPbPBvDsfyVjeHLeMxFzWsGf7LXrNhr0Pm5pmor0NU11waSEC/3avGmnGw7DxuAyiFrQJa3tTzs/6EzzMkNluWTGjKMwt5OKFVZwiXVAS4M548dsjmMHFcvQTKrmUWroOWYGbhy3lIWG/TujYpkxqyxxyozZhdmOrcl/xcalTV9OKqHy0qLim0ZpKcFqWL8wzESB3MqVA4wXws0KfMEKxq37B4G7hGj7yJfB6Pnw1TA6eNHde9Pexg55RB6TPonILtkj78k+GRHuHXin3mfvi3/of/K/+t82Ut9rcx6SC+Z//w2l4+BE</latexit><latexit sha1_base64="MbprTF7I+1x4HKW5VW+Vg1SCS6Q=">AAAC03icdVJbb9MwFHbCbYRbgUdejqiKWgmqBCENBEiTeOFxk+gWVFeV45601hw7i51uVYgEiFf+HG/8Bv4Ebpoh1sGRLH/nO9+5+JLkUhgbhj89/8rVa9dv7NwMbt2+c/de5/6DQ6PLguOIa6mLOGEGpVA4ssJKjPMCWZZIPEqO363jR0ssjNDqg13lOMnYXIlUcGYdNe38ogtmq7Tu0yXy6qweBL1tBiielGIJjX9aA+UzbaGNOvmGdgBzI6RW/fjpuXYAb4GaMptWrRyoUBDXkPbPBvDsfyVjeHLeMxFzWsGf7LXrNhr0Pm5pmor0NU11waSEC/3avGmnGw7DxuAyiFrQJa3tTzs/6EzzMkNluWTGjKMwt5OKFVZwiXVAS4M548dsjmMHFcvQTKrmUWroOWYGbhy3lIWG/TujYpkxqyxxyozZhdmOrcl/xcalTV9OKqHy0qLim0ZpKcFqWL8wzESB3MqVA4wXws0KfMEKxq37B4G7hGj7yJfB6Pnw1TA6eNHde9Pexg55RB6TPonILtkj78k+GRHuHXin3mfvi3/of/K/+t82Ut9rcx6SC+Z//w2l4+BE</latexit><latexit sha1_base64="MbprTF7I+1x4HKW5VW+Vg1SCS6Q=">AAAC03icdVJbb9MwFHbCbYRbgUdejqiKWgmqBCENBEiTeOFxk+gWVFeV45601hw7i51uVYgEiFf+HG/8Bv4Ebpoh1sGRLH/nO9+5+JLkUhgbhj89/8rVa9dv7NwMbt2+c/de5/6DQ6PLguOIa6mLOGEGpVA4ssJKjPMCWZZIPEqO363jR0ssjNDqg13lOMnYXIlUcGYdNe38ogtmq7Tu0yXy6qweBL1tBiielGIJjX9aA+UzbaGNOvmGdgBzI6RW/fjpuXYAb4GaMptWrRyoUBDXkPbPBvDsfyVjeHLeMxFzWsGf7LXrNhr0Pm5pmor0NU11waSEC/3avGmnGw7DxuAyiFrQJa3tTzs/6EzzMkNluWTGjKMwt5OKFVZwiXVAS4M548dsjmMHFcvQTKrmUWroOWYGbhy3lIWG/TujYpkxqyxxyozZhdmOrcl/xcalTV9OKqHy0qLim0ZpKcFqWL8wzESB3MqVA4wXws0KfMEKxq37B4G7hGj7yJfB6Pnw1TA6eNHde9Pexg55RB6TPonILtkj78k+GRHuHXin3mfvi3/of/K/+t82Ut9rcx6SC+Z//w2l4+BE</latexit><latexit sha1_base64="MbprTF7I+1x4HKW5VW+Vg1SCS6Q=">AAAC03icdVJbb9MwFHbCbYRbgUdejqiKWgmqBCENBEiTeOFxk+gWVFeV45601hw7i51uVYgEiFf+HG/8Bv4Ebpoh1sGRLH/nO9+5+JLkUhgbhj89/8rVa9dv7NwMbt2+c/de5/6DQ6PLguOIa6mLOGEGpVA4ssJKjPMCWZZIPEqO363jR0ssjNDqg13lOMnYXIlUcGYdNe38ogtmq7Tu0yXy6qweBL1tBiielGIJjX9aA+UzbaGNOvmGdgBzI6RW/fjpuXYAb4GaMptWrRyoUBDXkPbPBvDsfyVjeHLeMxFzWsGf7LXrNhr0Pm5pmor0NU11waSEC/3avGmnGw7DxuAyiFrQJa3tTzs/6EzzMkNluWTGjKMwt5OKFVZwiXVAS4M548dsjmMHFcvQTKrmUWroOWYGbhy3lIWG/TujYpkxqyxxyozZhdmOrcl/xcalTV9OKqHy0qLim0ZpKcFqWL8wzESB3MqVA4wXws0KfMEKxq37B4G7hGj7yJfB6Pnw1TA6eNHde9Pexg55RB6TPonILtkj78k+GRHuHXin3mfvi3/of/K/+t82Ut9rcx6SC+Z//w2l4+BE</latexit>
✏(X,Y, ~w) =X
~x2X
�f(x)� ~w · ~x
�2
<latexit sha1_base64="ho9Z9LUPpxjT2rX39OWcnvZHPpo=">AAAC5HicbVJNbxMxEPUuX2X5aKBHLiOioI1Uot0KCRBUqsSFYysRulUcIq/jTax67cX2hkar3LlwAMSVH8WN/8KhzmYrkaQjWfP85o09nnFaCG5sFP31/Bs3b92+s3M3uHf/wcPd1qPHH40qNWV9qoTSSUoME1yyvuVWsKTQjOSpYKfp+btl/HTGtOFKfrDzgg1zMpE845RYR41a/zp4SmyVLUI8Y7S6WHSDLQYw+1zyGdT7LwvAdKwsNFEnX9EBZoXhQskw2Yez/St1Fw4BmzIfVU0CYC4hcS7lkxCy8KILz68/upZ0Px0EnQSeXdXgKFytKbBzOOicbWjqk/EbnClNhIC125u8Uasd9aLaYBvEDWijxo5HrT94rGiZM2mpIMYM4qiww4poy6lgrgWlYQWh52TCBg5KkjMzrOopLaDjmDG4ctySFmr2/4yK5MbM89Qpc2KnZjO2JK+LDUqbvRpWXBalZZKuLspKAVbBcuQw5ppRK+YOEKq5qxXolGhCrfsYgWtCvPnkbdA/6L3uxScv2kdvm27soCfoKQpRjF6iI/QeHaM+ol7qffW+ez/8if/N/+n/Wkl9r8nZQ2vm/74Eyn7l0A==</latexit><latexit sha1_base64="ho9Z9LUPpxjT2rX39OWcnvZHPpo=">AAAC5HicbVJNbxMxEPUuX2X5aKBHLiOioI1Uot0KCRBUqsSFYysRulUcIq/jTax67cX2hkar3LlwAMSVH8WN/8KhzmYrkaQjWfP85o09nnFaCG5sFP31/Bs3b92+s3M3uHf/wcPd1qPHH40qNWV9qoTSSUoME1yyvuVWsKTQjOSpYKfp+btl/HTGtOFKfrDzgg1zMpE845RYR41a/zp4SmyVLUI8Y7S6WHSDLQYw+1zyGdT7LwvAdKwsNFEnX9EBZoXhQskw2Yez/St1Fw4BmzIfVU0CYC4hcS7lkxCy8KILz68/upZ0Px0EnQSeXdXgKFytKbBzOOicbWjqk/EbnClNhIC125u8Uasd9aLaYBvEDWijxo5HrT94rGiZM2mpIMYM4qiww4poy6lgrgWlYQWh52TCBg5KkjMzrOopLaDjmDG4ctySFmr2/4yK5MbM89Qpc2KnZjO2JK+LDUqbvRpWXBalZZKuLspKAVbBcuQw5ppRK+YOEKq5qxXolGhCrfsYgWtCvPnkbdA/6L3uxScv2kdvm27soCfoKQpRjF6iI/QeHaM+ol7qffW+ez/8if/N/+n/Wkl9r8nZQ2vm/74Eyn7l0A==</latexit><latexit sha1_base64="ho9Z9LUPpxjT2rX39OWcnvZHPpo=">AAAC5HicbVJNbxMxEPUuX2X5aKBHLiOioI1Uot0KCRBUqsSFYysRulUcIq/jTax67cX2hkar3LlwAMSVH8WN/8KhzmYrkaQjWfP85o09nnFaCG5sFP31/Bs3b92+s3M3uHf/wcPd1qPHH40qNWV9qoTSSUoME1yyvuVWsKTQjOSpYKfp+btl/HTGtOFKfrDzgg1zMpE845RYR41a/zp4SmyVLUI8Y7S6WHSDLQYw+1zyGdT7LwvAdKwsNFEnX9EBZoXhQskw2Yez/St1Fw4BmzIfVU0CYC4hcS7lkxCy8KILz68/upZ0Px0EnQSeXdXgKFytKbBzOOicbWjqk/EbnClNhIC125u8Uasd9aLaYBvEDWijxo5HrT94rGiZM2mpIMYM4qiww4poy6lgrgWlYQWh52TCBg5KkjMzrOopLaDjmDG4ctySFmr2/4yK5MbM89Qpc2KnZjO2JK+LDUqbvRpWXBalZZKuLspKAVbBcuQw5ppRK+YOEKq5qxXolGhCrfsYgWtCvPnkbdA/6L3uxScv2kdvm27soCfoKQpRjF6iI/QeHaM+ol7qffW+ez/8if/N/+n/Wkl9r8nZQ2vm/74Eyn7l0A==</latexit><latexit sha1_base64="ho9Z9LUPpxjT2rX39OWcnvZHPpo=">AAAC5HicbVJNbxMxEPUuX2X5aKBHLiOioI1Uot0KCRBUqsSFYysRulUcIq/jTax67cX2hkar3LlwAMSVH8WN/8KhzmYrkaQjWfP85o09nnFaCG5sFP31/Bs3b92+s3M3uHf/wcPd1qPHH40qNWV9qoTSSUoME1yyvuVWsKTQjOSpYKfp+btl/HTGtOFKfrDzgg1zMpE845RYR41a/zp4SmyVLUI8Y7S6WHSDLQYw+1zyGdT7LwvAdKwsNFEnX9EBZoXhQskw2Yez/St1Fw4BmzIfVU0CYC4hcS7lkxCy8KILz68/upZ0Px0EnQSeXdXgKFytKbBzOOicbWjqk/EbnClNhIC125u8Uasd9aLaYBvEDWijxo5HrT94rGiZM2mpIMYM4qiww4poy6lgrgWlYQWh52TCBg5KkjMzrOopLaDjmDG4ctySFmr2/4yK5MbM89Qpc2KnZjO2JK+LDUqbvRpWXBalZZKuLspKAVbBcuQw5ppRK+YOEKq5qxXolGhCrfsYgWtCvPnkbdA/6L3uxScv2kdvm27soCfoKQpRjF6iI/QeHaM+ol7qffW+ez/8if/N/+n/Wkl9r8nZQ2vm/74Eyn7l0A==</latexit>
✏(~w) =X
~x2X
�f(x)� ~w · ~x
�2
<latexit sha1_base64="zE7n0shh43T5GuaZnHuz21Scndo=">AAAC3nicbVJNj9MwEHXC12742AJHLiOqovRAlayQAMFKK3HhuEiULapL5biT1lrHCbZTtop62QsHQHvld3Hjj3DGTYNE2x3J8vObNzP2jJNCCmOj6LfnX7t+4+atvf3g9p279w5a9x98MHmpOfZ5LnM9SJhBKRT2rbASB4VGliUST5OzNyv/6Ry1Ebl6bxcFjjI2VSIVnFlHjVt/OnTGbJUuQzpHXp0vu8EOAxQ/l2IO9fnLEiif5BYar5Ov6YBiYYTMVdgQXTgCaspsXDVSoELBwG2JmIaQhuddeHp10lrS/XQYdAbw5F91R9FqQ0HdRoPOxy1NnZm+ommumZSwUb2JG7faUS+qDXZB3IA2aexk3PpFJzkvM1SWS2bMMI4KO6qYtoJLdI8vDRaMn7EpDh1ULEMzqur5LKHjmAm467ilLNTs/xEVy4xZZIlTZszOzLZvRV7lG5Y2fTGqhCpKi4qvC6WlBJvDatgwERq5lQsHGNfC3RX4jGnGrfsSgWtCvP3kXdA/7L3sxe+etY9fN93YI4/IYxKSmDwnx+QtOSF9wj3qXXjfvO8+87/6P/zLtdT3mpiHZMP8n38BYRbkSw==</latexit><latexit sha1_base64="zE7n0shh43T5GuaZnHuz21Scndo=">AAAC3nicbVJNj9MwEHXC12742AJHLiOqovRAlayQAMFKK3HhuEiULapL5biT1lrHCbZTtop62QsHQHvld3Hjj3DGTYNE2x3J8vObNzP2jJNCCmOj6LfnX7t+4+atvf3g9p279w5a9x98MHmpOfZ5LnM9SJhBKRT2rbASB4VGliUST5OzNyv/6Ry1Ebl6bxcFjjI2VSIVnFlHjVt/OnTGbJUuQzpHXp0vu8EOAxQ/l2IO9fnLEiif5BYar5Ov6YBiYYTMVdgQXTgCaspsXDVSoELBwG2JmIaQhuddeHp10lrS/XQYdAbw5F91R9FqQ0HdRoPOxy1NnZm+ommumZSwUb2JG7faUS+qDXZB3IA2aexk3PpFJzkvM1SWS2bMMI4KO6qYtoJLdI8vDRaMn7EpDh1ULEMzqur5LKHjmAm467ilLNTs/xEVy4xZZIlTZszOzLZvRV7lG5Y2fTGqhCpKi4qvC6WlBJvDatgwERq5lQsHGNfC3RX4jGnGrfsSgWtCvP3kXdA/7L3sxe+etY9fN93YI4/IYxKSmDwnx+QtOSF9wj3qXXjfvO8+87/6P/zLtdT3mpiHZMP8n38BYRbkSw==</latexit><latexit sha1_base64="zE7n0shh43T5GuaZnHuz21Scndo=">AAAC3nicbVJNj9MwEHXC12742AJHLiOqovRAlayQAMFKK3HhuEiULapL5biT1lrHCbZTtop62QsHQHvld3Hjj3DGTYNE2x3J8vObNzP2jJNCCmOj6LfnX7t+4+atvf3g9p279w5a9x98MHmpOfZ5LnM9SJhBKRT2rbASB4VGliUST5OzNyv/6Ry1Ebl6bxcFjjI2VSIVnFlHjVt/OnTGbJUuQzpHXp0vu8EOAxQ/l2IO9fnLEiif5BYar5Ov6YBiYYTMVdgQXTgCaspsXDVSoELBwG2JmIaQhuddeHp10lrS/XQYdAbw5F91R9FqQ0HdRoPOxy1NnZm+ommumZSwUb2JG7faUS+qDXZB3IA2aexk3PpFJzkvM1SWS2bMMI4KO6qYtoJLdI8vDRaMn7EpDh1ULEMzqur5LKHjmAm467ilLNTs/xEVy4xZZIlTZszOzLZvRV7lG5Y2fTGqhCpKi4qvC6WlBJvDatgwERq5lQsHGNfC3RX4jGnGrfsSgWtCvP3kXdA/7L3sxe+etY9fN93YI4/IYxKSmDwnx+QtOSF9wj3qXXjfvO8+87/6P/zLtdT3mpiHZMP8n38BYRbkSw==</latexit><latexit sha1_base64="zE7n0shh43T5GuaZnHuz21Scndo=">AAAC3nicbVJNj9MwEHXC12742AJHLiOqovRAlayQAMFKK3HhuEiULapL5biT1lrHCbZTtop62QsHQHvld3Hjj3DGTYNE2x3J8vObNzP2jJNCCmOj6LfnX7t+4+atvf3g9p279w5a9x98MHmpOfZ5LnM9SJhBKRT2rbASB4VGliUST5OzNyv/6Ry1Ebl6bxcFjjI2VSIVnFlHjVt/OnTGbJUuQzpHXp0vu8EOAxQ/l2IO9fnLEiif5BYar5Ov6YBiYYTMVdgQXTgCaspsXDVSoELBwG2JmIaQhuddeHp10lrS/XQYdAbw5F91R9FqQ0HdRoPOxy1NnZm+ommumZSwUb2JG7faUS+qDXZB3IA2aexk3PpFJzkvM1SWS2bMMI4KO6qYtoJLdI8vDRaMn7EpDh1ULEMzqur5LKHjmAm467ilLNTs/xEVy4xZZIlTZszOzLZvRV7lG5Y2fTGqhCpKi4qvC6WlBJvDatgwERq5lQsHGNfC3RX4jGnGrfsSgWtCvP3kXdA/7L3sxe+etY9fN93YI4/IYxKSmDwnx+QtOSF9wj3qXXjfvO8+87/6P/zLtdT3mpiHZMP8n38BYRbkSw==</latexit>
y = mx+ b<latexit sha1_base64="BBis2J/7LMdSRKDQenaOxGcSGig=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUA9C0YvHCsYW21A22027dHcTdjdiCP0XXjyoePXnePPfuG1z0NYHA4/3ZpiZFyacaeO6387C4tLyympprby+sbm1XdnZvddxqgj1Scxj1QqxppxJ6htmOG0limIRctoMh9djv/lIlWaxvDNZQgOB+5JFjGBjpYcMXSLxhI5R2K1U3Zo7AZonXkGqUKDRrXx1ejFJBZWGcKx123MTE+RYGUY4HZU7qaYJJkPcp21LJRZUB/nk4hE6tEoPRbGyJQ2aqL8nciy0zkRoOwU2Az3rjcX/vHZqovMgZzJJDZVkuihKOTIxGr+PekxRYnhmCSaK2VsRGWCFibEhlW0I3uzL88Q/qV3UvNvTav2qSKME+3AAR+DBGdThBhrgAwEJz/AKb452Xpx352PauuAUM3vwB87nD/wxj1o=</latexit><latexit sha1_base64="BBis2J/7LMdSRKDQenaOxGcSGig=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUA9C0YvHCsYW21A22027dHcTdjdiCP0XXjyoePXnePPfuG1z0NYHA4/3ZpiZFyacaeO6387C4tLyympprby+sbm1XdnZvddxqgj1Scxj1QqxppxJ6htmOG0limIRctoMh9djv/lIlWaxvDNZQgOB+5JFjGBjpYcMXSLxhI5R2K1U3Zo7AZonXkGqUKDRrXx1ejFJBZWGcKx123MTE+RYGUY4HZU7qaYJJkPcp21LJRZUB/nk4hE6tEoPRbGyJQ2aqL8nciy0zkRoOwU2Az3rjcX/vHZqovMgZzJJDZVkuihKOTIxGr+PekxRYnhmCSaK2VsRGWCFibEhlW0I3uzL88Q/qV3UvNvTav2qSKME+3AAR+DBGdThBhrgAwEJz/AKb452Xpx352PauuAUM3vwB87nD/wxj1o=</latexit><latexit sha1_base64="BBis2J/7LMdSRKDQenaOxGcSGig=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUA9C0YvHCsYW21A22027dHcTdjdiCP0XXjyoePXnePPfuG1z0NYHA4/3ZpiZFyacaeO6387C4tLyympprby+sbm1XdnZvddxqgj1Scxj1QqxppxJ6htmOG0limIRctoMh9djv/lIlWaxvDNZQgOB+5JFjGBjpYcMXSLxhI5R2K1U3Zo7AZonXkGqUKDRrXx1ejFJBZWGcKx123MTE+RYGUY4HZU7qaYJJkPcp21LJRZUB/nk4hE6tEoPRbGyJQ2aqL8nciy0zkRoOwU2Az3rjcX/vHZqovMgZzJJDZVkuihKOTIxGr+PekxRYnhmCSaK2VsRGWCFibEhlW0I3uzL88Q/qV3UvNvTav2qSKME+3AAR+DBGdThBhrgAwEJz/AKb452Xpx352PauuAUM3vwB87nD/wxj1o=</latexit><latexit sha1_base64="BBis2J/7LMdSRKDQenaOxGcSGig=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUA9C0YvHCsYW21A22027dHcTdjdiCP0XXjyoePXnePPfuG1z0NYHA4/3ZpiZFyacaeO6387C4tLyympprby+sbm1XdnZvddxqgj1Scxj1QqxppxJ6htmOG0limIRctoMh9djv/lIlWaxvDNZQgOB+5JFjGBjpYcMXSLxhI5R2K1U3Zo7AZonXkGqUKDRrXx1ejFJBZWGcKx123MTE+RYGUY4HZU7qaYJJkPcp21LJRZUB/nk4hE6tEoPRbGyJQ2aqL8nciy0zkRoOwU2Az3rjcX/vHZqovMgZzJJDZVkuihKOTIxGr+PekxRYnhmCSaK2VsRGWCFibEhlW0I3uzL88Q/qV3UvNvTav2qSKME+3AAR+DBGdThBhrgAwEJz/AKb452Xpx352PauuAUM3vwB87nD/wxj1o=</latexit>
1DInputs:
MinimizingSquaredError
Goal:pickweightsthatminimizesquarederror.
Approach#1:gradientdescent
Yourreadingderivedthisfor1Dinputs.
Doesthislookfamiliar?
MinimizingSquaredError
Goal:pickweightsthatminimizesquarederror.
Approach#2(therightway):analyticalsolution
• Thegradientis0attheerrorminimum.• Forlinearregression,thereisauniqueglobalminimumwithaclosedformula:
~w =⇣XTX
⌘�1XT~y
X ⌘⇥~x0 ~x1 . . . ~xn
⇤⌘
2
666664
1 1 . . . 1x00 x01 . . . x0n
x10 x11 . . . x1n...
.... . .
...xd0 xd1 . . . xdn
3
777775
ChangeofBasis
Polynomialregressionisjustlinearregressionwithachangeofbasis.
Performlinearregressiononthenewrepresentation.
2
6664
x0
x1...xd
3
7775�!
2
6666666664
x0
(x0)2
x1
(x1)2
...xd
(xd)2
3
7777777775
2
6664
x0
x1...xd
3
7775�!
2
6666666666666664
x0
(x0)2
(x0)3
x1
(x1)2
(x1)3
...xd
(xd)2
(xd)3
3
7777777777777775
quadraticbasis
cubicbasis
ChangeofBasisDemo
K-NearestNeighborsHypothesisSpace
Supervisedlearning• Foreveryinputinthedataset,weknowtheoutput
Classification• Outputsarediscrete• Categorylabels
Thelearnedmodel:• We’lltalkaboutthisinabit.
K-nearestneighborsalgorithmTraining:• Storeallofthetestpointsandtheirlabels.• Canuseadatastructurelikeakd-treethatspeedsuplocalizedlookup.
Prediction:• Findthektraininginputsclosesttothetestinput.• Outputthemostcommonlabelamongthem.
KNNimplementationdecisions• Howshouldwemeasuredistance?• (Euclideandistancebetweeninputvectors.)
• Whatifthere’satieforthenearestpoints?• (Includeallpointsthataretied.)
• Whatifthere’satieforthemost-commonlabel?• (Removethemost-distantpointuntilapluralityisachieved.)
• Whatifthere’satieforboth?• (Weneedsomearbitrarytie-breakingrule.)
(andpossibleanswers)
KNNHypothesisSpace
Whatdoesthelearnedmodellooklike?
Weightednearestneighbors• Idea:closerpointsshouldmattermore.
• Solution:weightthevoteby
• Insteadofcontributingonevoteforitslabel,eachneighborcontributes votesforitslabel.
Whydoweevenneedkneighbors?Idea:ifwe’reweightingbydistance,wecangivealltrainingpointsavote.• Pointsthatarefarawaywilljusthavereallysmallweight.
Whymightthisbeabadidea?• Slow:wehavetosumovereverypointinthetrainingset.• Ifwe’reusingakd-tree,wecangettheneighborsquicklyandsumoverasmallset.
Thesameideascanapplytoregression.• K-nearestneighborssetting:• Supervisedlearning(weknowthecorrectoutputforeachtestpoint).• Classification(smallnumberofdiscretelabels).
vs.
• Locally-weightedregressionsetting:• Supervisedlearning(weknowthecorrectoutputforeachtestpoint).• Regression(outputsarecontinuous).
Locally-WeightedAverage• Insteadoftakingamajorityvote,averagethey-values.
• Wecouldaverageovertheknearestneighbors.
• Wecouldweighttheaveragebydistance.
• Betteryet,doboth.
LocallyWeightedRegression
Keyidea:Foranypointwewanttopredict,computealinearregressionwitherrorweightedbydistance.
Asbefore,wefindthelinearfunctionthatminimizestotalerror,butweredefinetotalerror,sothatcloserpointscountmore:
X
~x2data
⇣y � f̂(~x)
⌘
dist(~xt, ~x)=
X
~x2data
✏2~x||~xt � ~x||2
SupervisedLearningPhases• Fitting(a.k.a.training)• Processdata• Createthemodelthatwillbeusedforprediction
• Prediction(a.k.a.testing)• Evaluatethemodelonnewinputs• Comparemodels
Describetheworkdoneineachphase:• Linearregression• KNN