I have implemented various optimizers (gradient descent, momentum, adam, etc.) based on gradient descent using only numpy not deep learning framework like TensorFlow.
2. 2018.12.15. MODUCON
• Research Scientist
• Ph. D. in Physics
• Research interests
• Generative models (GANs)
• Style transfer
• Reinforcement learning
• Generate and Transfer for Art (GTA Lab)
• github: https://github.com/ilguyi
• e-mail: ilgu.yi@modulabs.co.kr
8. 2018.12.15. MODUCON
Examples
• Portfolio optimization
• variables: amounts invested in different assets
• constraints: budget, max./min. investment per asset, minimum return
• objective: overall risk or return variance
• Device sizing in electronic circuits
• variables: device widths and lengths
• constraints: manufacturing limits, timing requirements, maximum area
• objective: power consumption
• Data fitting
• variables: model parameters
• constraints: prior information, parameter limits
• objective: measure of misfit or prediction error
Slide credit: Boyd & Vandenberghe, https://web.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf, p. 3
9. 2018.12.15. MODUCON
Why do We Care?
• Optimization is at the heart of many (most practical?) machine learning
algorithms
• Linear regression:
• Classification (logistic regression or SVM):
minimize
w
||Xw y||2
<latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit><latexit sha1_base64="FN7RoaBkQDz4822lh9I0IboJCYw=">AAACGXicbZDLSgMxFIYzXmu9VV26CRbBjWVGBF0W3HRZwV6gU0smPdMGk8yQZCx1Oq/hxldx40IRl7rybUwvC209EPj4/3PIOX8Qc6aN6347S8srq2vruY385tb2zm5hb7+uo0RRqNGIR6oZEA2cSagZZjg0YwVEBBwawd3V2G/cg9IskjdmGENbkJ5kIaPEWKlTcP1Edq0PJh1kqS+I6eswFUwywR4gy7CPR6PmAJ/i4Wh0m55lnULRLbmTwovgzaCIZlXtFD79bkQTAdJQTrRueW5s2ilRhlEOWd5PNMSE3pEetCxKIkC308llGT62SheHkbJPGjxRf0+kRGg9FIHtnKw+743F/7xWYsLLdspknBiQdPpRmHBsIjyOCXeZAmr40AKhitldMe0TRaixYeVtCN78yYtQPyt5lq/Pi+XKLI4cOkRH6AR56AKVUQVVUQ1R9Iie0St6c56cF+fd+Zi2LjmzmQP0p5yvH2bzocA=</latexit>
minimize
w
nX
i=1
log(1 + exp( yix>
i w))
<latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit><latexit sha1_base64="L8eYEYOwPDVTRuogWL6sJ5SRlfY=">AAACQXicbZBLTxsxFIU9PFpISxtg2Y1FhBSEQDMVEmyQkNiwBKmBSJkQeZw7wcKPkX2nJFjz19jwD9h13w0LqoptN3VCFryuZOvTOff6cbJCCodx/CuamZ2b//BxYbH26fPSl6/15ZVTZ0rLocWNNLadMQdSaGihQAntwgJTmYSz7PJw7J/9BOuE0T9wVEBXsYEWueAMg9Srt9NS94MP6K8qnyIM0eVeCS2UuIaqoilNXal6Xuwn1bnXQZBmQJs0oZs0hWHR3BoFs6LD8X4eTjBFRa82Nnr1RrwdT4q+hWQKDTKt4179Lu0bXirQyCVzrpPEBXY9syi4hKqWlg4Kxi/ZADoBNVPgun6SQEXXg9KnubFhaaQT9fmEZ8q5kcpCp2J44V57Y/E9r1Nivtf1QhclguZPF+WlpGjoOE7aFxY4ylEAxq0Ib6X8glnGMYRaCyEkr7/8Fk6/byeBT3YaB0fTOBbIN7JGmiQhu+SAHJFj0iKc3JDf5IH8iW6j++hv9PjUOhNNZ1bJi4r+/QdBgbEg</latexit>
or ||w||2
+ C
nX
i=1
⇠i s.t. ⇠i 1 yix>
i w, ⇠i 0
<latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit><latexit sha1_base64="6B03gjAYLYyPcCL21hjDtGuh5ls=">AAACZHicbVFda9swFJW9j3ZZt7krexqMy8JgsC3YpdC+DAp96WMHS1uIkyArciuqD0+6bhMU/8m99bEv+x2T3Qy2dBckzj3nXCQdFZUUDtP0NoofPX7ydGPzWe/51ouXr5Lt16fO1JbxITPS2POCOi6F5kMUKPl5ZTlVheRnxdVRq59dc+uE0d9xUfGxohdalIJRDNQ08TnyOVrljW0gh+XyZrmc+N0GPsER5K5WUy++Zs3E6yDPRehaWzfkSu8GOOj6P4rkPyCDL7Bo23m7TcIJpmrg5vO6L50m/XSQdgUPQbYCfbKqk2nyM58ZViuukUnq3ChLKxx7alEwyZteXjteUXZFL/goQE0Vd2PfhdTAh8DMoDQ2LI3QsX9PeKqcW6giOBXFS7euteT/tFGN5cHYC13VyDW7P6isJaCBNnGYCcsZykUAlFkR7grsklrKMPxLL4SQrT/5ITjdHWQBf9vrHx6v4tgkb8l78pFkZJ8ckmNyQoaEkbtoI0qi7ehXvBXvxG/urXG0mtkh/1T87jdCc7c+</latexit>
Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 5
10. 2018.12.15. MODUCON
We still Care…
• Maximum likelihood estimation:
• Collaborative filtering:
• k-means:
• And more (graphical models, feature selection, active learning, control)
Slide credit: Duchi, Convex Optimization for Machine Learning Fall 2009, p. 6
maximize
✓
nX
i=1
log p✓(xi)
<latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit><latexit sha1_base64="Z/lbKWlvpXmZKmvFJGbDo+uBlPY=">AAACNXicbVDLSgMxFM34tr6qLt0Ei6AbmRFBN4LgxoULBatCpw6Z9E4bTDJDckdah/kpN/6HK124UMStv2BaK/g6EHI45x6Se+JMCou+/+iNjI6NT0xOTVdmZufmF6qLS2c2zQ2HOk9lai5iZkEKDXUUKOEiM8BULOE8vjro++fXYKxI9Sn2Mmgq1tYiEZyhk6LqUZjrlvMBixA7gKx0N3TRJoViXaHEDZQlDWlocxUVYi8oLwvtBJm2aRZ9Zda7zis3omrN3/QHoH9JMCQ1MsRxVL0PWynPFWjkklnbCPwMmwUzKLiEshLmFjLGr1gbGo5qpsA2i8HWJV1zSosmqXFHIx2o3xMFU9b2VOwmFcOO/e31xf+8Ro7JbrMQOssRNP98KMklxZT2K6QtYYCj7DnCuBHur5R3mGEcXZEVV0Lwe+W/5GxrM3D8ZLu2fzisY4qskFWyTgKyQ/bJITkmdcLJLXkgz+TFu/OevFfv7XN0xBtmlskPeO8fLriuBg==</latexit>
minimize
w
X
i j
log 1 + exp(w>
xi w>
xj)
<latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit><latexit sha1_base64="TUfyFrghYNVb4IwaYImAjY4SsM4=">AAACWHicbVDLahsxFNVMkyZxX26yzOZSU3AoDTOl0C4D3WSZQpwELNdo5Du2Ej0G6U5jd5ifLHTR/ko3kR0v8jogcTjnHj1OUWkVKMv+Jumzjc3nW9s7nRcvX71+0327exZc7SUOpNPOXxQioFYWB6RI40XlUZhC43lx9W3pn/9EH5Szp7SocGTE1KpSSUFRGncdr+0k+kjNddtwwjmFsjHKKqN+YdsCBx5qM24U8HiwhMsoaTeNG5bUz+EDcJxX/esfMeyqFuZxtIWPcFe4bA+AezWd0cG428sOsxXgMcnXpMfWOBl3f/OJk7VBS1KLEIZ5VtGoEZ6U1Nh2eB2wEvJKTHEYqRUGw6hZFdPC+6hMoHQ+LkuwUu8mGmFCWJgiThpBs/DQW4pPecOayq+jRtmqJrTy9qKy1kAOli3DRMWuSC8iEdKr+FaQM+GFpNh1J5aQP/zyY3L26TCP/Pvn3tHxuo5tts/esT7L2Rd2xI7ZCRswyf6w/8lGspn8S1m6le7cjqbJOrPH7iHdvQGLSrTH</latexit>
minimize
µ1,··· ,µk
J(µ) =
kX
j=1
X
i2Cj
||xi µj||2
<latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit><latexit sha1_base64="5D+ThTq6ib1eZT+eFi8T0Aj2c4A=">AAACZHicbVDPSxwxGM1Mf2i3th0rPRVK6FKw0MqMFPQiCF6kJwtdFTbrkMlkNG6SGZIv4prNP9lbj73072h2dw6t9oOQl/fe9yV5VSeFhTz/maSPHj95urb+bPB848XLV9nm61PbOsP4iLWyNecVtVwKzUcgQPLzznCqKsnPqunRQj+74caKVn+HWccnil5q0QhGIVJl5onTddQ5eKJc6YvwCRNWt2DjviCmIXgC/BZs45XQQok7HgIm+Ot21D/iA0ysU6W/PijCRXT3R4GJ0Pgo8tE8n99GJuDPq5HXYT6/8LuhzIb5Tr4s/BAUPRiivk7K7AepW+YU18AktXZc5B1MPDUgmORhQJzlHWVTesnHEWqquJ34ZUgBf4hMjZvWxKUBL9m/OzxV1s5UFZ2KwpW9ry3I/2ljB83+xAvdOeCarS5qnMTQ4kXiuBaGM5CzCCgzIr4VsytqKIOY+yCGUNz/8kNwurtTRPzty/DwuI9jHb1F79E2KtAeOkTH6ASNEEO/krUkSzaT3+lGupW+WVnTpO/ZQv9U+u4PpT65Pg==</latexit>
25. 2018.12.15. MODUCON
Computing the Gradient
• Use backpropagation to compute gradients efficiently
• Need a differentiable function
• Can’t use functions like argmax or hard binary
• Unless using a different way to compute gradients
Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19
26. 2018.12.15. MODUCON
How to Pick the Learning Rate?
• Too big = diverge, too small = slow convergence
• No “one learning rate to rule them all”
• Start from a high value and keep cutting by half if model diverges
• Learning rate schedule: decay learning rate over time
Slide credit: P. Ramachandran, CS 598 LAZ- Cutting-Edge Trends in Deep Learning and Recognition, Lec05, p. 19
27. 2018.12.15. MODUCON
Too Small Learning Rate
Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112
28. 2018.12.15. MODUCON
Too Large Learning Rate
Figure credit: A. Géron, Hands-on Machine Learning with Scikit-Learn & TensorFlow, chap 1, p. 112
29. 2018.12.15. MODUCON
Learning Rate
• Which is better?
• Is it better to keep learning rate?
• Decay learning rate
appropriately
Figure credit: cs231n spring 2018 slide: Lecture 6. p. 84
30. 2018.12.15. MODUCON
Stochastic Gradient Descent
• Gradient Descent
• Cross entropy error, CEE
w0
k := wk ⌘
@L
@wk
• Loss
(mini-batch)
Loss
• Mini-batch size:
•
L
1
N i
yi log ˆyi
L
1
m i
yi log ˆyi
m<latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit><latexit sha1_base64="qWck30ONVt2kTy0KbJVkwBgxduc=">AAAB6HicbZBNSwMxEIZn/az1q+rRS7AInsquCHoseOmxBfsB7VKy6Wwbm2SXJCuU0l/gxYMiXv1J3vw3pu0etPWFwMM7M2TmjVLBjfX9b29jc2t7Z7ewV9w/ODw6Lp2ctkySaYZNlohEdyJqUHCFTcutwE6qkcpIYDsa38/r7SfUhifqwU5SDCUdKh5zRq2zGrJfKvsVfyGyDkEOZchV75e+eoOEZRKVZYIa0w381IZTqi1nAmfFXmYwpWxMh9h1qKhEE04Xi87IpXMGJE60e8qShft7YkqlMRMZuU5J7cis1ubmf7VuZuO7cMpVmllUbPlRnAliEzK/mgy4RmbFxAFlmrtdCRtRTZl12RRdCMHqyevQuq4Ejhs35Wotj6MA53ABVxDALVShBnVoAgOEZ3iFN+/Re/HevY9l64aXz5zBH3mfP9ddjPc=</latexit>
32. 2018.12.15. MODUCON
The Momentum Method
• Introduce velocity variable:
• It is the direction and speed at which parameters move through parameter
space
• Momentum is mass times velocity term in physics
• The momentum algorithm assumes unit mass
• A hyperparameter determines exponential decay
v<latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit><latexit sha1_base64="235vjU4tS6ea5yNRrUD4VlxqA8o=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8NJjC/YD2lA220m7drMJu5tCCf0FXjwo4tWf5M1/47bNQVtfWHh4Z4adeYNEcG1c99spbG3v7O4V90sHh0fHJ+XTs7aOU8WwxWIRq25ANQousWW4EdhNFNIoENgJJg+LemeKSvNYPppZgn5ER5KHnFFjreZ0UK64VXcpsgleDhXI1RiUv/rDmKURSsME1brnuYnxM6oMZwLnpX6qMaFsQkfYsyhphNrPlovOyZV1hiSMlX3SkKX7eyKjkdazKLCdETVjvV5bmP/VeqkJ7/2MyyQ1KNnqozAVxMRkcTUZcoXMiJkFyhS3uxI2pooyY7Mp2RC89ZM3oX1T9Sw3byu1eh5HES7gEq7BgzuoQR0a0AIGCM/wCm/Ok/PivDsfq9aCk8+cwx85nz/lAY0A</latexit>
2 [0, 1)<latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit><latexit sha1_base64="Ba3q1rx4knV/3kRC2rXIsDTuCFk=">AAAB+nicbZDLSgMxFIbP1Futt6ku3QSLoCBlRgRdFtx0WcFeYKaUTJppQ5PMkGSUMvZR3LhQxK1P4s63Mb0stPWHwMd/zuGc/FHKmTae9+0U1tY3NreK26Wd3b39A7d82NJJpghtkoQnqhNhTTmTtGmY4bSTKopFxGk7Gt1O6+0HqjRL5L0Zp7Qr8ECymBFsrNVzy+EAC4FRyCQKvAvkn/fcilf1ZkKr4C+gAgs1eu5X2E9IJqg0hGOtA99LTTfHyjDC6aQUZpqmmIzwgAYWJRZUd/PZ6RN0ap0+ihNlnzRo5v6eyLHQeiwi2ymwGerl2tT8rxZkJr7p5kymmaGSzBfFGUcmQdMcUJ8pSgwfW8BEMXsrIkOsMDE2rZINwV/+8iq0Lqu+5burSq2+iKMIx3ACZ+DDNdSgDg1oAoFHeIZXeHOenBfn3fmYtxacxcwR/JHz+QPBV5Je</latexit>
53. 2018.12.15. MODUCON
Learning Rate is Crucial
• Learning rate: most difficult hyperparameters to set
• It significantly affects model performance
• Loss function is highly sensitive to some directions in parameter space and
insensitive to others
• Momentum helps but introduces another hyperparameters
• If direction of sensitivity is axis aligned, separate learning rate for each
parameter and adjust them throughput learning
55. 2018.12.15. MODUCON
Adagrad
• J. Duchi, et. al., Adaptive subgradient methods for online learning and
stochastic optimization (http://jmlr.org/papers/v12/duchi11a.html)
• It adapts the learning rate to the parameters, performing smaller updates (i.e.
low learning rates) for parameters associated with frequently occurring
features, and larger updates (i.e. high learning rates) for parameters
associated with infrequent features
• Previously, we performed an update for all parameters at once as every
parameter used the same learning rate
• As Adagrad uses a different learning rate for every parameter at every time
step
w<latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit><latexit sha1_base64="E83TDbJMZtue5+b0Rpx/6FdLIkA=">AAAB8XicbVDLSsNAFL3xWeur6tLNYBFclUQEXRbcdFnBPrANZTKdtEMnkzBzo5TQv3DjQhG3/o07/8ZJm4W2Hhg4nHMvc+4JEikMuu63s7a+sbm1Xdop7+7tHxxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJbe53Hrk2Ilb3OE24H9GREqFgFK300I8ojoMwe5oNKlW35s5BVolXkCoUaA4qX/1hzNKIK2SSGtPz3AT9jGoUTPJZuZ8anlA2oSPes1TRiBs/myeekXOrDEkYa/sUkrn6eyOjkTHTKLCTeUKz7OXif14vxfDGz4RKUuSKLT4KU0kwJvn5ZCg0ZyinllCmhc1K2JhqytCWVLYleMsnr5L2Zc2z/O6qWm8UdZTgFM7gAjy4hjo0oAktYKDgGV7hzTHOi/PufCxG15xi5wT+wPn8Af2EkSE=</latexit>
wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit>
⌘<latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit><latexit sha1_base64="dCG1lAu4MamMLQGuw8Bo7bnQR6c=">AAAB63icbZBNSwMxEIazftb6VfXoJVgET2VXBD0WvPRYwX5Au5RsOtuGJtklmRVK6V/w4kERr/4hb/4bs+0etPWFwMM7M2TmjVIpLPr+t7exubW9s1vaK+8fHB4dV05O2zbJDIcWT2RiuhGzIIWGFgqU0E0NMBVJ6EST+7zeeQJjRaIfcZpCqNhIi1hwhrnVB2SDStWv+QvRdQgKqJJCzUHlqz9MeKZAI5fM2l7gpxjOmEHBJczL/cxCyviEjaDnUDMFNpwtdp3TS+cMaZwY9zTShft7YsaUtVMVuU7FcGxXa7n5X62XYXwXzoROMwTNlx/FmaSY0PxwOhQGOMqpA8aNcLtSPmaGcXTxlF0IwerJ69C+rgWOH26q9UYRR4mckwtyRQJyS+qkQZqkRTgZk2fySt485b14797HsnXDK2bOyB95nz8Kso4+</latexit>
wi<latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit><latexit sha1_base64="0prmBy7BzaaAl9QhvxDl6sMPGkU=">AAAB9XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXBTZcV7APasWTSTBuayQxJxlKG+Q83LhRx67+482/MtLPQ1gOBwzn3ck+OHwuujeN8o9LG5tb2Tnm3srd/cHhUPT7p6ChRlLVpJCLV84lmgkvWNtwI1osVI6EvWNef3uV+94kpzSP5YOYx80IyljzglBgrPQ5CYiZ+kM6yYcqzYbXm1J0F8DpxC1KDAq1h9WswimgSMmmoIFr3XSc2XkqU4VSwrDJINIsJnZIx61sqSci0ly5SZ/jCKiMcRMo+afBC/b2RklDreejbyTylXvVy8T+vn5jg1ku5jBPDJF0eChKBTYTzCvCIK0aNmFtCqOI2K6YTogg1tqiKLcFd/fI66VzVXcvvr2uNZlFHGc7gHC7BhRtoQBNa0AYKCp7hFd7QDL2gd/SxHC2hYucU/gB9/gBMFJMJ</latexit>
t<latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit><latexit sha1_base64="pA2axXYPJdYvt8IEY8q8iokWeXo=">AAAB6HicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3GSZgHlAMoSeTk3SpudBd40QQr7AjQtF3PpJ7vwbO8ksNPFCw+FWFV11g1RJQ6777Wxsbm3v7Bb2ivsHh0fHpZPTlkkyLbApEpXoTsANKhljkyQp7KQaeRQobAfj+3m9/YTayCR+oEmKfsSHsQyl4GStBvVLZbfiLsTWwcuhDLnq/dJXb5CILMKYhOLGdD03JX/KNUmhcFbsZQZTLsZ8iF2LMY/Q+NPFojN2aZ0BCxNtX0xs4f6emPLImEkU2M6I08is1ubmf7VuRuGdP5VxmhHGYvlRmClGCZtfzQZSoyA1scCFlnZXJkZcc0E2m6INwVs9eR1a1xXPcuOmXK3lcRTgHC7gCjy4hSrUoA5NEIDwDK/w5jw6L86787Fs3XDymTP4I+fzB+H5jP4=</latexit>