zero-order convex optimization: an introduction

Zero-order Convex Optimization: An Introduction

Ziniu [email protected]

The Chinese University of Hong Kong, Shenzhen, Shenzhen, China

Nov. 28, 2020

Mainly based on the paper:

Duchi, John C., et al. ”Optimal rates for zero-order convex optimization: The power of two function evaluations.” IEEE

Transactions on Information Theory 61.5 (2015): 2788-2806.

Outline

Introduction to Zero-order Optimization

Key Results

Smooth Optimization

Non-smooth Optimization

Lower Bounds

Proofs

Proof of Theorem 1

Proof of Proposition 1

Conclusion

Introduction to Zero-order Optimization 2 / 73


I We consider the following optimization:

minx∈X

f(x).

I When f is convex and importantly differentiable, many first-order (i.e., gradient-based)

methods can be applied [Nesterov, 2018].

– Typically, the convergence rate is dimension-free.

I However, if f is non-differentiable and only zero-order information is available?

– We have access to f(x) but not ∇f(x).– Even ∇f(x) could not be properly defined.


ZO Application: Adversarial Attack

I Imagine there is a hacker who wants to attack the trained neural nets.

– He can send a query to the “black-box” model and get the feedback.

I The objective to find some adversarial examples that incurs large losses:

minε∈Rd

−L(f(x0 + ε), y), s.t. ||ε|| ≤ δ.

f<latexit sha1_base64="7QQfQx4iNX04F8go9Nvv8s3hbAI=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFkUxGUL9gG1SDKd1qHTJGQmQin6A27128Q/0L/wzpiCWkQnJDlz7j1n5t4bJlIo7XmvBWdhcWl5pbhaWlvf2Nwqb++0VJyljDdZLOO0EwaKSxHxphZa8k6S8mAcSt4OR+cm3r7jqRJxdKUnCe+Ng2EkBoIFmqjG4KZc8aqeXe488HNQQb7qcfkF1+gjBkOGMTgiaMISARQ9XfjwkBDXw5S4lJCwcY57lEibURanjIDYEX2HtOvmbER746msmtEpkt6UlC4OSBNTXkrYnObaeGadDfub99R6mrtN6B/mXmNiNW6J/Us3y/yvztSiMcCprUFQTYllTHUsd8lsV8zN3S9VaXJIiDO4T/GUMLPKWZ9dq1G2dtPbwMbfbKZhzZ7luRnezS1pwP7Pcc6D1lHV96p+47hSO8tHXcQe9nFI8zxBDZeoo2m9H/GEZ+fCkY5yss9Up5BrdvFtOQ8fQFyPaw==</latexit><latexit sha1_base64="7QQfQx4iNX04F8go9Nvv8s3hbAI=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFkUxGUL9gG1SDKd1qHTJGQmQin6A27128Q/0L/wzpiCWkQnJDlz7j1n5t4bJlIo7XmvBWdhcWl5pbhaWlvf2Nwqb++0VJyljDdZLOO0EwaKSxHxphZa8k6S8mAcSt4OR+cm3r7jqRJxdKUnCe+Ng2EkBoIFmqjG4KZc8aqeXe488HNQQb7qcfkF1+gjBkOGMTgiaMISARQ9XfjwkBDXw5S4lJCwcY57lEibURanjIDYEX2HtOvmbER746msmtEpkt6UlC4OSBNTXkrYnObaeGadDfub99R6mrtN6B/mXmNiNW6J/Us3y/yvztSiMcCprUFQTYllTHUsd8lsV8zN3S9VaXJIiDO4T/GUMLPKWZ9dq1G2dtPbwMbfbKZhzZ7luRnezS1pwP7Pcc6D1lHV96p+47hSO8tHXcQe9nFI8zxBDZeoo2m9H/GEZ+fCkY5yss9Up5BrdvFtOQ8fQFyPaw==</latexit><latexit sha1_base64="7QQfQx4iNX04F8go9Nvv8s3hbAI=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFkUxGUL9gG1SDKd1qHTJGQmQin6A27128Q/0L/wzpiCWkQnJDlz7j1n5t4bJlIo7XmvBWdhcWl5pbhaWlvf2Nwqb++0VJyljDdZLOO0EwaKSxHxphZa8k6S8mAcSt4OR+cm3r7jqRJxdKUnCe+Ng2EkBoIFmqjG4KZc8aqeXe488HNQQb7qcfkF1+gjBkOGMTgiaMISARQ9XfjwkBDXw5S4lJCwcY57lEibURanjIDYEX2HtOvmbER746msmtEpkt6UlC4OSBNTXkrYnObaeGadDfub99R6mrtN6B/mXmNiNW6J/Us3y/yvztSiMcCprUFQTYllTHUsd8lsV8zN3S9VaXJIiDO4T/GUMLPKWZ9dq1G2dtPbwMbfbKZhzZ7luRnezS1pwP7Pcc6D1lHV96p+47hSO8tHXcQe9nFI8zxBDZeoo2m9H/GEZ+fCkY5yss9Up5BrdvFtOQ8fQFyPaw==</latexit><latexit sha1_base64="7QQfQx4iNX04F8go9Nvv8s3hbAI=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFkUxGUL9gG1SDKd1qHTJGQmQin6A27128Q/0L/wzpiCWkQnJDlz7j1n5t4bJlIo7XmvBWdhcWl5pbhaWlvf2Nwqb++0VJyljDdZLOO0EwaKSxHxphZa8k6S8mAcSt4OR+cm3r7jqRJxdKUnCe+Ng2EkBoIFmqjG4KZc8aqeXe488HNQQb7qcfkF1+gjBkOGMTgiaMISARQ9XfjwkBDXw5S4lJCwcY57lEibURanjIDYEX2HtOvmbER746msmtEpkt6UlC4OSBNTXkrYnObaeGadDfub99R6mrtN6B/mXmNiNW6J/Us3y/yvztSiMcCprUFQTYllTHUsd8lsV8zN3S9VaXJIiDO4T/GUMLPKWZ9dq1G2dtPbwMbfbKZhzZ7luRnezS1pwP7Pcc6D1lHV96p+47hSO8tHXcQe9nFI8zxBDZeoo2m9H/GEZ+fCkY5yss9Up5BrdvFtOQ8fQFyPaw==</latexit>

x<latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit>

f(x)<latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit>

x1<latexit sha1_base64="aRygnQNe64d88SptSZeB5T52ETg=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+2y9XnKqjlz0LXAMqMKuRlF9wiQESMOSIwBFDEg7hIaOnCxcOUuJ6mBAnCAU6znGPEmlzyuKU4RE7pu+Idl3DxrRXnplWMzolpFeQ0sYeaRLKE4TVabaO59pZsb95T7Snutsd/X3jFRErcUXsX7pp5n91qhaJIY51DQHVlGpGVceMS667om5uf6lKkkNKnMIDigvCTCunfba1JtO1q956Ov6mMxWr9szk5nhXt6QBuz/HOQtaB1XXqbpnh5XaiRl1ETvYxT7N8wg11NFAk7xHeMQTnq26FVu5dfOZahWMZhvflvXwAQ4LkCE=</latexit><latexit sha1_base64="aRygnQNe64d88SptSZeB5T52ETg=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+2y9XnKqjlz0LXAMqMKuRlF9wiQESMOSIwBFDEg7hIaOnCxcOUuJ6mBAnCAU6znGPEmlzyuKU4RE7pu+Idl3DxrRXnplWMzolpFeQ0sYeaRLKE4TVabaO59pZsb95T7Snutsd/X3jFRErcUXsX7pp5n91qhaJIY51DQHVlGpGVceMS667om5uf6lKkkNKnMIDigvCTCunfba1JtO1q956Ov6mMxWr9szk5nhXt6QBuz/HOQtaB1XXqbpnh5XaiRl1ETvYxT7N8wg11NFAk7xHeMQTnq26FVu5dfOZahWMZhvflvXwAQ4LkCE=</latexit><latexit sha1_base64="aRygnQNe64d88SptSZeB5T52ETg=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+2y9XnKqjlz0LXAMqMKuRlF9wiQESMOSIwBFDEg7hIaOnCxcOUuJ6mBAnCAU6znGPEmlzyuKU4RE7pu+Idl3DxrRXnplWMzolpFeQ0sYeaRLKE4TVabaO59pZsb95T7Snutsd/X3jFRErcUXsX7pp5n91qhaJIY51DQHVlGpGVceMS667om5uf6lKkkNKnMIDigvCTCunfba1JtO1q956Ov6mMxWr9szk5nhXt6QBuz/HOQtaB1XXqbpnh5XaiRl1ETvYxT7N8wg11NFAk7xHeMQTnq26FVu5dfOZahWMZhvflvXwAQ4LkCE=</latexit><latexit sha1_base64="aRygnQNe64d88SptSZeB5T52ETg=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+2y9XnKqjlz0LXAMqMKuRlF9wiQESMOSIwBFDEg7hIaOnCxcOUuJ6mBAnCAU6znGPEmlzyuKU4RE7pu+Idl3DxrRXnplWMzolpFeQ0sYeaRLKE4TVabaO59pZsb95T7Snutsd/X3jFRErcUXsX7pp5n91qhaJIY51DQHVlGpGVceMS667om5uf6lKkkNKnMIDigvCTCunfba1JtO1q956Ov6mMxWr9szk5nhXt6QBuz/HOQtaB1XXqbpnh5XaiRl1ETvYxT7N8wg11NFAk7xHeMQTnq26FVu5dfOZahWMZhvflvXwAQ4LkCE=</latexit>

f(x1)<latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit>

x2<latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit>

f(x2)<latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit>

xt<latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit>

f(xt1)<latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit>

· · · · · ·<latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit>Introduction to Zero-order Optimization 4 / 73


I The hacker can only adopt a ZO algorithm to optimize the adversarial example.


x<latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit><latexit sha1_base64="MyiEmFnntf+qNJvnSMVFXu6341g=">AAACxHicjVHLSsNAFD2Nr/quunQTLIKrkoigy6IgLluwD6hFknRaQ6dJmJmIpegPuNVvE/9A/8I74xTUIjohyZlz7zkz994w47FUnvdacObmFxaXissrq2vrG5ulre2mTHMRsUaU8lS0w0AyHiesoWLFWTsTLBiFnLXC4ZmOt26ZkHGaXKpxxrqjYJDE/TgKFFH1u+tS2at4ZrmzwLegDLtqaekFV+ghRYQcIzAkUIQ5Akh6OvDhISOuiwlxglBs4gz3WCFtTlmMMgJih/Qd0K5j2YT22lMadUSncHoFKV3skyalPEFYn+aaeG6cNfub98R46ruN6R9arxGxCjfE/qWbZv5Xp2tR6OPE1BBTTZlhdHWRdclNV/TN3S9VKXLIiNO4R3FBODLKaZ9do5Gmdt3bwMTfTKZm9T6yuTne9S1pwP7Pcc6C5mHF9yp+/ahcPbWjLmIXezigeR6jigvU0DDej3jCs3PucEc6+WeqU7CaHXxbzsMHaxyPfQ==</latexit>

f(x)<latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit><latexit sha1_base64="WXGLFgYhcn7uwDFZnhTpniap+X8=">AAACx3icjVHLTsJAFD3UF+ILdemmkZjghrTGRJdEN7rDRJAEiWmHARpKp5lOCYS48Afc6p8Z/0D/wjtjSVRidJq2Z86958zce/04DBLlOK85a2FxaXklv1pYW9/Y3Cpu7zQSkUrG60yEQjZ9L+FhEPG6ClTIm7Hk3tAP+Y0/ONfxmxGXSSCiazWJeXvo9aKgGzBPaapbHh/eFUtOxTHLngduBkrIVk0UX3CLDgQYUgzBEUERDuEhoacFFw5i4tqYEicJBSbOcY8CaVPK4pThETugb492rYyNaK89E6NmdEpIrySljQPSCMqThPVptomnxlmzv3lPjae+24T+fuY1JFahT+xfulnmf3W6FoUuTk0NAdUUG0ZXxzKX1HRF39z+UpUih5g4jTsUl4SZUc76bBtNYmrXvfVM/M1kalbvWZab4l3fkgbs/hznPGgcVVyn4l4dl6pn2ajz2MM+yjTPE1RxgRrq5N3HI57wbF1awhpZ489UK5dpdvFtWQ8fkRqQUg==</latexit>


f(x1)<latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit><latexit sha1_base64="eOX1OywNhHoWC8J/CVQhLocynYU=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcde7dTLFkVSy1zGtgalKBXLS6+4AJdxPCQIQRDBEE4gIOUnjZsWEiIu8SYOE7IV3GGexRIm1EWowyH2AF9+7RrazaivfRMldqjUwJ6OSlN7JAmpjxOWJ5mqnimnCX7m/dYecq7jejvaq+QWIErYv/STTL/q5O1CPRwqGrwqaZEMbI6T7tkqivy5uaXqgQ5JMRJ3KU4J+wp5aTPptKkqnbZW0fF31SmZOXe07kZ3uUtacD2z3FOg8ZexbYq9tl+qXqkR53HFrZRpnkeoIoT1FAn72s84gnPxqlxYwyNu89UI6c1m/i2jIcPNcGQ9g==</latexit>

x2<latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit><latexit sha1_base64="s6zvf+7s2l4TomASIkk3KrnnZ3A=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIi6LLopsuK9gG1lGQ6rUPTJEwmaimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y/OdLx1w2UiovBSTWLeHXvDUAwE8xRRF3e9Sq9YcsqOWfY8cDNQQrbqUfEFV+gjAkOKMThCKMIBPCT0dODCQUxcF1PiJCFh4hz3KJA2pSxOGR6xI/oOadfJ2JD22jMxakanBPRKUto4IE1EeZKwPs028dQ4a/Y376nx1Heb0N/PvMbEKlwT+5dulvlfna5FYYATU4OgmmLD6OpY5pKaruib21+qUuQQE6dxn+KSMDPKWZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wHzUrZdcru+VGpepqNOo897OOQ5nmMKmqoo0HeQzziCc9WzQqt1Lr9TLVymWYX35b18AEQa5Ai</latexit>

f(x2)<latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit><latexit sha1_base64="iMRqU4/SZDcaJKvXl4dJQ0YquX0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5IUQZdFN4KbCvYBtZQkndbYNImTibQWV/6AW/0x8Q/0L7wzTkEtohOSnDn3njNz73XjwE+EZb1mjLn5hcWl7HJuZXVtfSO/uVVPopR7rOZFQcSbrpOwwA9ZTfgiYM2YM2foBqzhDk5kvHHLeOJH4YUYx6w9dPqh3/M9RxBV7xVHnfJ+J1+wSpZa5iywNShAr2qUf8EluojgIcUQDCEE4QAOEnpasGEhJq6NCXGckK/iDPfIkTalLEYZDrED+vZp19JsSHvpmSi1R6cE9HJSmtgjTUR5nLA8zVTxVDlL9jfvifKUdxvT39VeQ2IFroj9SzfN/K9O1iLQw5GqwaeaYsXI6jztkqquyJubX6oS5BATJ3GX4pywp5TTPptKk6jaZW8dFX9TmZKVe0/npniXt6QB2z/HOQvq5ZJtlezzg0LlWI86ix3sokjzPEQFp6iiRt7XeMQTno0z48YYGXefqUZGa7bxbRkPHzgikPc=</latexit>

xt<latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit><latexit sha1_base64="MqyuasQuSS02sTy89z3CWhjrIGA=">AAACyHicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl0I64qmFaopSTTaR2aJmEyUUvpxh9wq18m/oH+hXfGFNQiOiHJmXPvOTP33iAJRaoc57Vgzc0vLC4Vl0srq2vrG+XNrUYaZ5Jxj8VhLK8CP+WhiLinhAr5VSK5PwxC3gwGpzrevOUyFXF0qUYJbw/9fiR6gvmKKO++M1aTTrniVB2z7Fng5qCCfNXj8guu0UUMhgxDcERQhEP4SOlpwYWDhLg2xsRJQsLEOSYokTajLE4ZPrED+vZp18rZiPbaMzVqRqeE9EpS2tgjTUx5krA+zTbxzDhr9jfvsfHUdxvRP8i9hsQq3BD7l26a+V+drkWhh2NTg6CaEsPo6ljukpmu6JvbX6pS5JAQp3GX4pIwM8ppn22jSU3ture+ib+ZTM3qPctzM7zrW9KA3Z/jnAWNg6rrVN2Lw0rtJB91ETvYxT7N8wg1nKEOj7wFHvGEZ+vcSqw7a/SZahVyzTa+LevhA0j6kXA=</latexit>

f(xt1)<latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit><latexit sha1_base64="CfPsrjUtphTHo4yz82QEYAW7zJI=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkWoC0sigi6LbtxZwT6wlpJMp3VomoRkIpZat/6AW/0t8Q/0L7wzpqAW0QlJzpx7z5m597qhJ2JpWa8ZY2Z2bn4hu5hbWl5ZXcuvb9TiIIkYr7LAC6KG68TcEz6vSiE93ggj7gxcj9fd/omK1294FIvAv5DDkLcGTs8XXcEcSdRlt3jbHsk9e7zbzheskqWXOQ3sFBSQrkqQf8EVOgjAkGAADh+SsAcHMT1N2LAQEtfCiLiIkNBxjjFypE0oi1OGQ2yfvj3aNVPWp73yjLWa0SkevREpTeyQJqC8iLA6zdTxRDsr9jfvkfZUdxvS3029BsRKXBP7l26S+V+dqkWiiyNdg6CaQs2o6ljqkuiuqJubX6qS5BASp3CH4hFhppWTPptaE+vaVW8dHX/TmYpVe5bmJnhXt6QB2z/HOQ1q+yXbKtnnB4XycTrqLLawjSLN8xBlnKKCKnn7eMQTno0zIzHujPvPVCOTajbxbRkPH6ORkrc=</latexit>

· · · · · ·<latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit>



f(xt)<latexit sha1_base64="/howoMVXh5bLlJ0I8MONH+IMzh0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcdsdsplqyKpZY5DWwNStCrFhdfcIEuYnjIEIIhgiAcwEFKTxs2LCTEXWJMHCfkqzjDPQqkzSiLUYZD7IC+fdq1NRvRXnqmSu3RKQG9nJQmdkgTUx4nLE8zVTxTzpL9zXusPOXdRvR3tVdIrMAVsX/pJpn/1claBHo4VDX4VFOiGFmdp10y1RV5c/NLVYIcEuIk7lKcE/aUctJnU2lSVbvsraPibypTsnLv6dwM7/KWNGD75zinQWOvYlsV+2y/VD3So85jC9so0zwPUMUJaqiT9zUe8YRn49S4MYbG3WeqkdOaTXxbxsMH1SSROQ==</latexit><latexit sha1_base64="/howoMVXh5bLlJ0I8MONH+IMzh0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcdsdsplqyKpZY5DWwNStCrFhdfcIEuYnjIEIIhgiAcwEFKTxs2LCTEXWJMHCfkqzjDPQqkzSiLUYZD7IC+fdq1NRvRXnqmSu3RKQG9nJQmdkgTUx4nLE8zVTxTzpL9zXusPOXdRvR3tVdIrMAVsX/pJpn/1claBHo4VDX4VFOiGFmdp10y1RV5c/NLVYIcEuIk7lKcE/aUctJnU2lSVbvsraPibypTsnLv6dwM7/KWNGD75zinQWOvYlsV+2y/VD3So85jC9so0zwPUMUJaqiT9zUe8YRn49S4MYbG3WeqkdOaTXxbxsMH1SSROQ==</latexit><latexit sha1_base64="/howoMVXh5bLlJ0I8MONH+IMzh0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcdsdsplqyKpZY5DWwNStCrFhdfcIEuYnjIEIIhgiAcwEFKTxs2LCTEXWJMHCfkqzjDPQqkzSiLUYZD7IC+fdq1NRvRXnqmSu3RKQG9nJQmdkgTUx4nLE8zVTxTzpL9zXusPOXdRvR3tVdIrMAVsX/pJpn/1claBHo4VDX4VFOiGFmdp10y1RV5c/NLVYIcEuIk7lKcE/aUctJnU2lSVbvsraPibypTsnLv6dwM7/KWNGD75zinQWOvYlsV+2y/VD3So85jC9so0zwPUMUJaqiT9zUe8YRn49S4MYbG3WeqkdOaTXxbxsMH1SSROQ==</latexit><latexit sha1_base64="/howoMVXh5bLlJ0I8MONH+IMzh0=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkWom5KIoMuiG8FNBfuAKiVJpzU2LycTaS2u/AG3+mPiH+hfeGecglpEJyQ5c+49Z+be6yaBnwrLes0ZM7Nz8wv5xcLS8srqWnF9o5HGGfdY3YuDmLdcJ2WBH7G68EXAWglnTugGrOkOjmW8ect46sfRuRgl7DJ0+pHf8z1HENXolYcdsdsplqyKpZY5DWwNStCrFhdfcIEuYnjIEIIhgiAcwEFKTxs2LCTEXWJMHCfkqzjDPQqkzSiLUYZD7IC+fdq1NRvRXnqmSu3RKQG9nJQmdkgTUx4nLE8zVTxTzpL9zXusPOXdRvR3tVdIrMAVsX/pJpn/1claBHo4VDX4VFOiGFmdp10y1RV5c/NLVYIcEuIk7lKcE/aUctJnU2lSVbvsraPibypTsnLv6dwM7/KWNGD75zinQWOvYlsV+2y/VD3So85jC9so0zwPUMUJaqiT9zUe8YRn49S4MYbG3WeqkdOaTXxbxsMH1SSROQ==</latexit>

xt<latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit>

xt<latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit><latexit sha1_base64="JC6ZDsH+f5NNaQ5unFUZXsKppR8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl002VF+4BaSjKd1tC8mEx8UAR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/vhUxdvXXGRBEl/Iu5T3Im8UB8OAeZKo89u+7JcrTtXRy54FrgEVmNVIyi+4xAAJGHJE4IghCYfwkNHThQsHKXE9TIgThAId57hHibQ5ZXHK8Igd03dEu65hY9orz0yrGZ0S0itIaWOPNAnlCcLqNFvHc+2s2N+8J9pT3e2O/r7xioiVuCL2L9008786VYvEEMe6hoBqSjWjqmPGJdddUTe3v1QlySElTuEBxQVhppXTPttak+naVW89HX/TmYpVe2Zyc7yrW9KA3Z/jnAWtg6rrVN2zw0rtxIy6iB3sYp/meYQa6migSd4jPOIJz1bdiq3cuvlMtQpGs41vy3r4AK0rkGQ=</latexit>


· · · · · ·<latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit><latexit sha1_base64="Ysi0g3sRaNbJEQpZWvpQExx8djs=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRIRdFl047IF+4C2SDKd1qFpJiQTpRTFrT/gVv9K/AP9C+9MU1CL6ITMnDn3njNz5/pRIBLlOG85a2FxaXklv1pYW9/Y3Cpu7zQSmcaM15kMZNzyvYQHIuR1JVTAW1HMvZEf8KY/PNfx5g2PEyHDSzWOeHfkDULRF8xTRHU6rCdVMp2viiWn7JhhzwM3AyVkoyqLr+igBwmGFCNwhFCEA3hI6GvDhYOIuC4mxMWEhIlz3KFA2pSyOGV4xA5pHtCunbEh7bVnYtSMTgnoj0lp44A0kvJiwvo028RT46zZ37wnxlPfbUyrn3mNiFW4JvYv3Szzvzpdi0Ifp6YGQTVFhtHVscwlNa+ib25/qUqRQ0Scxj2Kx4SZUc7e2TaaxNSu39Yz8XeTqVm9Z1luig99S2qw+7Od86BxVHadsls7LlXOslbnsYd9HFI/T1DBBaqok3eEJzzjxapZt9a99TBNtXKZZhffhvX4CcfAlGU=</latexit>



More Applications of Zero-order Optimization

I Bandit Optimization [Flaxman et al., 2005, Bartlett et al., 2008, Agarwal et al., 2010].

I Simulation-based optimization [Spall, 2005].

I Graphical model inference [Wainwright and Jordan, 2008].

I Policy optimization [Wierstra et al., 2014, Salimans et al., 2017].

I Escaping the local minimum in ERM [Jin et al., 2018].


Main Difficulty of Zero-Order Optimization

I (The curse of dimension) Convergence rate of ZO methods scales up with dimension d

[Duchi et al., 2012b, Jamieson et al., 2012, Shamir, 2013, Duchi et al., 2015].

I Consider to optimize a Lipschitz continuous function f : |f(x)− f(y)| ≤ L||x− y|| with

only zero-order information.

– The lower bound of total evaluation numbers suggests an exponential dependence on d.

Lower Bound:

⌊L

2ε

⌋d– The simple method of grid search is minimax optimal!

Upper Bound:

(⌊L

2ε

⌋+ 1

)d


Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Key Results 9 / 73

A General Start: Stochastic Optimization

I We need to restrict our attention to not-so-hard class: convex function class.

minθ∈Θ

f(θ) := EP [F (θ;X)] =

∫XF (θ;x) dP (x). (1)

where Θ ⊆ Rd is a compact convex set, P is a distribution over X and for every x ∈ X we

have F (·;x) is closed and convex.

I Each iteration, we have access to F (θ;x) by drawing x from P (this process is not

controlled by algorithms).

– In machine learning, x is a training sample, Fi(θ;x) is the individual loss and f(θ) is the

population/empirical loss.

– We do not know ∇f(θ) or even ∇F (θ;x).

Key Results 10 / 73

Intuition of Zero-order Optimization

I We can utilize multiple function evaluations to approximate the directional derivative:

F ′(θ; z, x) = limu↓0

1

u(F (θ + uz;x)− F (θ;x)) = 〈∇F (θ;x), z〉.

I In high-level, zero-order algorithms sample a noisy gradient to optimize.

1

u(F (θ + uz;x)− F (θ;x)) z ≈ zz>∇F (θ;x).

where u > 0 is a small perturbation size and z is a random vector.

I Taking the expectation on both sides and with the assumption that E[zz>

]= Id, we

obtain an estimate of ∇F (θ;x).

Key Results 11 / 73

Algorithmic Assumptions

I We consider a mirror descent type algorithm:

θt+1 = argminθ∈Θ

⟨gt, θ

⟩+

1

α(t)Dψ

(θ, θt

), (2)

– α(t)∞t=1 is a non-increasing sequence of step sizes.

– gt ∈ Rd is a (subgradient) vector.

– Dψ is a Bregman distance defined by the proximal function ψ:

Dψ(θ, τ) := ψ(θ)− ψ(τ)− 〈∇ψ(τ), θ − τ〉.

Key Results 12 / 73


Assumption 1.

The proximal function ψ is 1-strongly convex with respect to the norm || · ||. The domain Θ is

compact and there exists R <∞ such that Dψ(θ∗, θ) ≤ 12R

2 for θ ∈ Θ.

I If we consider || · || as `2-norm, ψ(θ) = 12 ||θ||22 and Θ = Rn, we have

Dψ(θ, τ) = 12 ‖θ − τ‖

22, and,

θt+1 = argminθ∈Θ

⟨gt, θ

⟩+

1

α(t)Dψ

(θ, θt

)= θt − α(t)gt.

Key Results 13 / 73


Assumption 2.

There is a constant G <∞ such that the (sub)gradient g satisfies that E[||g(θ;X)||2

]≤ G2 for

all θ ∈ Θ.

I The variance of (sub)gradient is controlled by G.

I This holds when F (·;x) are G-Lipschitz continuous with respect to the norm || · ||.

Key Results 14 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Key Results 15 / 73

Main Idea

I The directional gradient estimate can approximate the gradient:

Gsm(θ;u, z, x) :=F (θ + uz;x)− F (θ;x)

uz, (3)

E [Gsm(θ;u, z, x)] = ∇f(θ) + u · bias, (4)

here we assume x ∼ P (x) and z ∼ µ(z) and the bias term will be shown later.

I We use a noisy gradient estimate gt and shrink the parameter u to control the bias.

gt = Gsm

(θt;ut, Z

t, Xt)

=F (θt + utZ

t;Xt)− F (θt;Xt)

utZt. (5)

Key Results 16 / 73

More Assumptions about Smooth Optimization

I Different from stochastic mirror descent, zero-order algorithms need to ensure the parameter

domain is well-defined.

Assumption 3.

The domain of Functions F and support of µ satisfies

domF (·;x) ⊃ Θ + u suppµ for x ∈ X .

and,

Eµ[ZZ>

]= Id.

Key Results 17 / 73


Assumption 4.

For Z ∼ µ, the quantity M(µ) =

√E[‖Z‖4 ‖Z‖2∗

]is finite. Moreover, there is a function

s : N→ R+ such that

E[‖〈g, Z〉Z‖2∗

]≤ s(d)‖g‖2∗ for any vector g ∈ Rd. (6)

I For example, µ is a standard Gaussian distribution N (0, Id) and || · || is the `2-norm.

I M(µ) =

√E[‖Z‖6

]=√

15d6 . d3.

I E[||〈g, Z〉Z||22

]= E

[g>ZZ>ZZ>g

]= g>E [(2 + d)I] g =⇒ s(d) . d.

Key Results 18 / 73


Assumption 5.

There is a function L : X → R+ such that for P -almost every x ∈ X , the function F (·;x) has

L(x)-Lipschitz continuous gradient with respect to the norm || · || and moreover the quantity

L(P ) :=

√E[(L(X))

2]

is finite.

Key Results 19 / 73

Gradient Approximation

Lemma 1.

Under Assumption 4 and 5, the gradient estimate (3) has the expectation:

E [Gsm(θ;u, Z,X)] = ∇f(θ) + uL(P )v(θ, u), (7)

for a vector v = v(θ, u) such that ||v||∗ ≤ 12E[||Z||2||Z||∗

]. Its expected squared norm has the

bound

E[‖Gsm(θ;u, Z,X)‖2∗

]≤ 2s(d)E

[‖g(θ;X)‖2∗

]+

1

2u2L(P )2M(µ)2. (8)

Here g(θ;x) ∈ ∂F (θ;x) is a subgradient with E [g(θ;x)] ∈ ∂f(θ).

Key Results 20 / 73

Implication of Lemma 1

I The estimate gt is unbiased up to a correction term of u.

E [Gsm(θ;u, Z,X)] = ∇f(θ) + uL(P )v(θ, u).

I The second moment is also unbiased up to an order u2t correction–within a factor s(d).


]≤ 2s(d)E

[‖g(θ;X)‖2∗

]+

1

2u2L(P )2M(µ)2.

I As long as we shrink ut, we can obtain arbitrary accurate estimates of the directional

derivative.

Key Results 21 / 73

Proof of Lemma 1: Preliminary

I We start with a general convex function h with Lh-Lipschitz continuous gradient w.r.t the

norm || · ||.I For any u > 0, we have that

h′(θ, z) =〈∇h(θ), uz〉

u≤ h(θ + uz)− h(θ)

u≤ 〈∇h(θ), uz〉+ (Lh/2) ‖uz‖2

u

= h′(θ, z) +Lhu

2‖z‖2,

I Therefore for any z ∈ Rd, we have that

h(θ + uz)− h(θ)

uz = h′(θ, z)z +

Lhu

2‖z‖2γ(u, θ, z)z, (9)

where γ is some function with range contained in [0, 1].

Key Results 22 / 73

Proof of Lemma 1: Preliminary

I By our assumption that E[ZZ>

]= Id, (9) implies that

E[h(θ + uZ)− h(θ)

uZ

]= E

[h′(θ, Z)Z +

Lhu

2‖Z‖2γ(u, θ, Z)Z

](10)

= E [〈∇h(θ), Z〉Z] + E[Lhu

2‖Z‖2γ(u, θ, Z)Z

](11)

= ∇h(θ) + uLhv(θ, u), (12)

where v(θ, u) ∈ Rd is an error vector with ||v(θ, u)||∗ ≤ 12E[‖Z‖2 ‖Z‖∗

].

Key Results 23 / 73

Proof of Lemma 1: The First Moment

I Recalling the gradient estimate in (7), expression (12) implies that

E [Gsm(θ;u, Z, x)] = ∇F (θ;x) + uL(x)v(θ, u), (13)

for some vector v = v(θ, u) with 2||v||∗ ≤ E[‖Z‖2 ‖Z‖∗

].

I Now taking the expectation over X, for the first term we have E [∇F (θ;X)] = ∇f(θt).

For the second term, by Jensen’s inequality we have that

E [L(X)‖v(θ, u)‖∗] ≤√

E [L(X)2] ‖v‖∗ ≤1

2L(P )E

[‖Z‖2‖Z‖∗

],

from which the bound (7) follows.

Key Results 24 / 73

Proof of Lemma 1: The Second Moment

I Applying (9) to F (·;X), we obtain that

Gsm(θ;u, Z,X) = 〈g(θ;X), Z〉Z +L(X)u

2‖Z‖2γZ,

for some function γ ≡ γ(u, θ, Z,X) ∈ [0, 1].

I To upper bound the second moment, we use the relation (a+ b)2 ≤ 2a2 + 2b2:


]≤ E

[(‖〈g(θ,X), Z〉Z‖∗ +

1

2‖L(X)u‖Z

∥∥2γZ∥∥∗

)2]

≤ 2E[‖〈g(θ,X), Z〉Z‖2∗

]+u2

2E[L(X)2‖Z‖4‖Z‖2∗

]≤ 2s(d)E

[‖g(θ;X)‖2∗

]+

1

2u2L(P )2M(µ)2.

Key Results 25 / 73

Key Result For Smooth Optimization

Theorem 1.

Under Assumption 1, 2 3, 4 and 5, consider a sequence θt∞t=1 generated by the mirror descent

update (2) using the gradient estimator (5), with step and perturbation parameter

α(t) = αR

2G√s(d)√t

and ut = uG√s(d)

L(P )M(µ)· 1

tfor t = 1, 2, . . .

Then for all k,

E[f(θ(k))− f (θ∗)

]≤ 2

RG√s(d)√k

maxα, α−1

+ αu2RG

√s(d)

k+ u

RG√s(d) log(2k)

k,

(14)

where θ(k) = 1k

∑kt=1 θ

t and the expectation is taken w.r.t. samples X and Z.

Key Results 26 / 73

Implication of Theorem 1

I We first compare the result with stochastic mirror descent with first-order information.

Method Step Size Perturbation Size Optimality Gap

First-order α√t

O(

1√k

)Zero-order α R

2G√s(d)√t

uG√s(d)

L(P )M(µ) · 1t O

(√s(d)√k

)I The convergence rate only slows down by

√s(d)!

– If we consider µ a Gaussian distribution over Rd or a uniform distribution over `2-ball,

s(d) . d.

– This is partially because we have to use a small step size in ZO algorithms.

Key Results 27 / 73

Implication of Theorem 1

I We see that a small perturbation size is applied to control the bias.

I Variance-control can also be achieved by multiple independent samples Zt,i, i = 1, · · · ,mto construct a more accurate gradient estimate.

gt =1

m

m∑i=1

Gsm(θt;ut, Zt,i, Xt).

I In this way, we may achieve a standard RG/√k convergence rate (see the next page).

Key Results 28 / 73

Smooth Optimization with Multiple Function Evaluations

Corollary 1.

Let Zt,i, i = 1, · · · ,m be sampled independently according to µ and at each iteration of mirror

descent use the gradient estimate gt = 1m

∑mi=1 Gsm(θt;ut, Z

t,i, Xt) with the step and

perturbation sizes

α(t) = αR

2Gmax√d/m, 1

· 1√t

and ut = uG

L(P )d3/2· 1

t.

There exists a universal constant C ≤ 5 such that for all k,

E[f(θ(k))− f (θ∗)

]≤ CRG

√1 + d/m√k

[max

α, α−1

+ αu2 1√

k+ u

log(2k)

k

].

Key Results 29 / 73

More Cooments on Corollary 1 and Theorem 1

I We could achieve a “dimension-free” result by choose m ≈ d in Corollary 1.

I However, if we define the sample complexity as the total number of function evaluations,

the sample complexity is clearly not dimension-free.

– There is a (currently unknown) trade-off about how many evaluations to apply.

I In high dimensional scenarios, we can properly choose the proximal function and the norm

|| · || to release the true power of mirror descent.

– Refer to [Duchi et al., 2015, Corollary 3] and [Beck and Teboulle, 2003].

I The term of maxα, α−1 is said to be robust in stochastic optimization.

– i.e., if we specify α wrongly, the final result is not so bad (ref to [Nemirovski et al., 2009]).

Key Results 30 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Key Results 31 / 73

Difficulty in Non-smooth Optimization

I Recall that by L-Lipschitz continuous gradient of F (·;x), we have


]≤ 2s(d)E

[‖g(θ;X)‖2∗

]+

1

2u2L(P )2M(µ)

2

. s(d)︸︷︷︸. d

E[‖g(θ;X)‖2∗

]︸︷︷︸≤G2

(u ∝

√s(d)E [‖g(θ;X)‖2∗]L(P )M(µ)

)

I For non-smooth case, we only have G-Lipschitz continuity, we have that


]≤ E

[∥∥∥∥F (θ + uZ;x)− F (θ;x)

uZ

∥∥∥∥2

2

]≤ G2 E

[‖Z‖42

]︸︷︷︸≥d2

.

I That is, the O(d2) term for non-smooth in contrast to the O(d) term for the smooth case.

Key Results 32 / 73

Solution: Smoothing the Non-smooth Functions

I For a general function f(θ), we can define the smoothed objective function,

fu(θ) := E[f(θ + uZ)] =

∫f(θ + uz) dµ(z).

I If f is Lipschitz continuous and convex, we can show that fu(θ) is differentiable even

though f is not [Duchi et al., 2012a, Nesterov and Spokoiny, 2017].

I Implication: if we smooth the non-smooth function F (θ;x) slightly, we may achieve a

convergence rate that is roughly the same as that in smooth case.

Key Results 33 / 73

Gradient Estimate for Non-smooth Functions

I Based on the above intuition, we can construct the gradient estimate:

Gns (θ;u1, u2, z1, z2, x) :=F (θ + u1z1 + u2z2;x)− F (θ + u1z1;x)

u2z2. (15)

Here z1, z2 are independently drawn from distributions µ1 and µ2 and u1,t∞t=1 and

u2,t∞t=1 are two positive non-increasing sequences with u2,t ≤ u1,t.

I And similarly, we have

gt =F (θt + u1,tZ

t1 + u2,tZ

t2;Xt)− F (θt + u1,tZ

t1;Xt)

u2,tZt2, (16)

where Zt1 serves the “smoothing” function and Zt2 severs the gradient estimate function.

Key Results 34 / 73

More Assumptions For Non-smooth Functions

Assumption 6.

There is a function G : X → R+ such that for every x ∈ X , the function F (·;x) is

G(x)-Lipschitz with respect to the `2-norm || · || and the quantity G(P ) =√E [G(X)2] is finite.

Key Results 35 / 73

More Assumptions For Non-smooth Functions

Assumption 7.

The smoothing distributions are one of the following pairs:

I both µ1 and µ2 are standard normal distribution in Rd.

I both µ1 and µ2 are uniform on the `2-ball of radius√d+ 2.

I µ1 is uniformly on the `2-ball of radius√d+ 2 and µ2 is uniform on the `2-sphere of radius√

d.

In addition, the domain of F (·;x) is well defined:

domF (·;x) ⊃ Θ + u1,1 suppµ1 + u2,1 suppµ2 for x ∈ X .

Key Results 36 / 73

Gradient Approximation For Non-smooth Functions

Lemma 2.

Under Assumption 6 and 7, the gradient estimator (15) has the expectation:

E [Gns (θ;u1, u2, Z1, Z2, X)] = ∇fu1(θ) +

u2

u1G(P )v (θ, u1, u2) , (17)

where v = v(θ, u1, u2) has bound ‖v‖2 ≤ 12E[‖Z2‖32

]. There exists a universal constant c such

that

E[‖Gns (θ;u1, u2, Z1, Z2, X)‖22

]≤ cG(P )2d

(√u2

u1d+ 1 + log d

). (18)

Key Results 37 / 73

Comments on Lemma 2

I Compared to Lemma 1, Lemma 2 suggests that the gradient estimate is nearly unbiased.

I If we could choose a small u2 such that√

u2

u1d is almost negligible, then we recover the

convergence rate of smooth optimization.

I Actually, there still is an additional log d term but it is expected to remove this in future

works.

Key Results 38 / 73

Convergence Result For Non-smooth Optimization

Theorem 2.

Under Assumption 1, 6 and 7, consider a sequence θt∞t=1 generated according to mirror

descent update 2 using the gradient estimator (16) with step and perturbation sizes

α(t) = αR

G(P )√d log(2d)

√t, u1,t = u

R

t, and u2,t = u

R

d2t2.

Then there exists a universal constant c such that for all k,

E[f(θ(k))− f (θ∗)

]≤ cmax

α, α−1

RG(P )√d log(2d)√k

+ cuRG(P )√d

log(2k)

k, (19)

where θ(k) = 1k

∑kt=1 θ

t and the expectation is taken w.r.t. samples X and Z.

Key Results 39 / 73

Comments on Theorem 2

I Compared to Theorem 1, Theorem 2 suggests that two-point zero-order algorithms for

non-smooth functions is at worst a factor√

log d worse than the rate for smooth functions.

Key Results 40 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Key Results 41 / 73

Minimax Error and Minimax Optimal

I Let F be a collection of pairs (F, P ), each of which defines a problem instance (1).

I Let Ak denote the collection of all algorithms that receives a sequences (Y1, · · · , Yk), each

of which contains two-point evaluations:

Y t =[F (θt, Xt), F (τ t, Xt)

].

Here (θt, τ t) can be determined by the algorithm.

I Given an algorithm A ∈ Ak and a pair (F, P ) ∈ F , the optimality gap is defined as

εk(A, F, P,Θ) := f(θ(k))− infθ∈Θ

f(θ) = EP [F (θ(k);X)]− infθ∈Θ

EP [F (θ;X)],

where θ(k) is the output of algorithm A at iteration k.

Key Results 42 / 73

Minimax Error and Minimax Optimal

I The minimax error is defined as

ε∗k(F ,Θ) := infA∈Ak

sup(F,P )∈F

E [εk(A, F, P,Θ)] , (20)

where expectation is taken over the observations (Y 1, · · · , Y k) and any additional

randomness in A.

I An algorithm A is called minimax optimal if its upper bound matches the lower bound up

to constant and logarithmic terms.

Key Results 43 / 73

Lower Bound For Two-point Evaluations

I For a given `p-norm || · ||p, we consider the class of linear functionals:

FG,p :=

(F, P ) | F (θ;x) = 〈θ, x〉 with EP[‖X‖2p

]≤ G2

.

I Each of which satisfies Assumption 6 (i.e., Lipschitz continuity).

I Moreover, ∇F (·;x) has Lipschitz constant 0 for all x.

I We consider the domain is equal to some `q-ball of radius, i.e.,

Θ =θ ∈ Rd

∣∣ ‖θ‖q ≤ R .

Key Results 44 / 73

Lower Bound For Two-point Evaluations

Proposition 1.

For the class FG,2 and Θ =θ ∈ Rd

∣∣ ‖θ‖q ≤ R, we have

ε∗k (FG,2,Θ) ≥ 1

12

(1− 1

q

)GR√k

mind1−1/q, k1−1/q

. (21)

I For k ≥ d, this lower bound translates to Ω(GR√kd1−1/q

).

Key Results 45 / 73

Comments on Lower Bound 1

I For q ≥ 2, the `2-ball of radius d1/2−1/qR contains the `q-ball of radius R, so the upper

bound in Theorem 1 and 2 be analyzed here.

I In particular, we have the upper bound that

RG√d√

k≤ RG

√dd1/2−1/q

√k

=RGd1−1/q

√k

.

I This implies that the algorithm for smooth optimization is optimal up to constant factors

and the algorithm for non-smooth optimization is also tight to within logarithmic factors.

Key Results 46 / 73

Lower Bound For Multiple Evaluations

I An inspection of the proof of Proposition 1 yields that

ε∗k (FG,2,Θ) ≥ 1

10

(1− 1

q

)GR√mk

mind1−1/q, k1−1/q

. (22)

I In Corollary 1, we have the upper bound O(RG

√d/m√k

).

I This indicates that when m→ d, the algorithm also achieves minimax optimal.

Key Results 47 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Proofs 48 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Proofs 49 / 73

Proof of Theorem 1

I By mirror descent with Assumption 1, we have that [Beck and Teboulle, 2003, Nemirovski

et al., 2009, Shalev-Shwartz, 2012]:

t∑t=1

f(θt)− f(θ∗) ≤k∑t=1

⟨gt, θt − θ∗

⟩≤ 1

2α(k)R2 +

k∑t=1

α(t)

2

∥∥gt∥∥2

∗ . (23)

I Now let’s introduce the error vector et := ∇f(θt)− gt,

k∑t=1

(f(θt)− f (θ∗)

)≤

k∑t=1

⟨gt, θt − θ∗

⟩+

k∑t=1

⟨et, θt − θ∗

⟩≤ 1

2α(k)R2 +

k∑t=1

α(t)

2

∥∥gt∥∥2

∗ +k∑t=1

⟨et, θt − θ∗

⟩.

(24)

Proofs 50 / 73

Proof of Theorem 1

I For the second moment term, by (8) in Lemma 1, we have that

E[∥∥gt∥∥2

∗

]≤ 2s(d)G2 +

1

2u2tL(P )2M(µ)2. (25)

We can properly choose ut ∝√s(d)G

L(P )M(µ) to control this term.

I For the last term in (24), by (7) in Lemma 1, we have that

k∑t=1

E[⟨et, θt − θ∗

⟩]≤ L(P )

k∑t=1

utE[‖vt‖∗

∥∥θt − θ∗∥∥] ≤ 1

2M(µ)RL(P )

k∑t=1

ut. (26)

The last step we use the relation ||θt − θ∗|| ≤√

2Dψ(θ∗, θ) ≤ R.

Proofs 51 / 73

Proof of Theorem 1

I By combing the above inequalities, we have that

t∑t=1

f(θt)− f(θ∗)

≤ R2

2α(k)+ s(d)G2

k∑t=1

α(t) +L(P )2M(µ)2

4

k∑t=1

u2tα(t) +

M(µ)RL(P )

2

k∑t=1

ut.

I It remains to plug-in the chosen step and perturbation sizes and to apply Jensen’s inequality.

Proofs 52 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Proofs 53 / 73


I Main idea: reduce the optimization to binary hypothesis testing problems.

I First, we construct a finite set of functions, upon of which the optimality gap is lower

bounded by the sign difference.

I Consequently, we lower bound the probability of sign difference after observing k random

samples with total variation distance by Le Cam’s inequality.

I Finally, we present a sharp bound of total variation distance for this problem.

Proofs 54 / 73

Proof of Proposition 1: The First Part

I We consider the binary vector v in the Boolean hypercube V = −1, 1d.

I The objective functions are in the form F (θ;x) = 〈θ, x〉.I For each v, Pv is the Gaussian distribution N (δv, σ2I), where δ > 0 is to be chosen later.

I Now, the problem becomes that

minθ∈Θ

fv(θ) := EPv[F (θ;X)] = δ〈θ, v〉, (27)

where Θ = θ ∈ Rd∣∣ ||θ||q ≤ R.

I It’s clear that the optimal solution is given by θv = −Rd1/qv.

Proofs 55 / 73


I We claim that for any θ ∈ Rd the optimality gap is bounded by (see the next page).

fv(θ)− fv (θv) ≥ 1− 1/q

d1/qδR

d∑j=1

1

sign(θj

)6= sign

(θvj). (28)

I To understand (28), we note that if sign(θj

)= sign

(θvj)

for all j, then (28) holds trivially.

Therefore, we only need to care the case where there exist some coordinates j such that

sign(θj

)6= sign

(θvj).

Proofs 56 / 73


I Let’s split θv the coordinates into two parts: I+ = vi = 1 and I− = vi = −1. Now,

we represent θv as below (only signs are shown):

θv =

(+, · · · ,+︸︷︷︸I−

∣∣∣∣ − · · · ,−︸︷︷︸I+

).

I With the same order, we can also represent the estimator θ as below (only signs are shown):

θ =

(−︸︷︷︸I−−

, · · · ,+︸︷︷︸I+−

∣∣∣∣ +︸︷︷︸I++

· · · ,−︸︷︷︸I−+

).

Here I−− and I++ are two “error” sets, in which the sign of θ is different from θv.

Proofs 57 / 73


I We define two optimization problems to lower bound the cost due to sign difference.

min v>θ, s.t. ‖θ‖q ≤ 1 (29)

min v>θ, s.t. ‖θ‖q ≤ 1; θj ≤ 0,∀j ∈ I−− ; θj ≥ 0,∀j ∈ I++ . (30)

I Denote the optimal solution by θA and θB , respectively. We have

θA = d−1/q(1I− − 1I+

), and θB = (d− c)−1/q

(1I+− − 1I−+

),

where 1A denotes the vector with 1 for coordinates are in A and 0 otherwise. In addition, c

is the sum of cardinalities of I−− and I++ .

Proofs 58 / 73


I As a consequence, the objective values are given:

v>θA = −d1−1/q, and v>θB = −(d− c)1−1/q.

I We use the fact that the function f(x) = −x1−1/q is convex for q ∈ [1,∞):

−∇f(d)c ≤ f(d− c)− f(d) =⇒ 1− 1/q

d1/qc ≤ −(d− c)1−1/q − (−d1−1/q)

=⇒ 1− 1/q

d1/qc ≤ v>θB − v>θA.

I Note that θB is the “optimal” estimator among all estimators with sign differences. Hence,

the above bound gives relation (28).

Proofs 59 / 73

Proof of Proposition 1: The Second Part

I We consider the performance on the mixture distribution P := (1/|V|)∑v∈V Pv, then,

maxv

EPv

[fv(θ)− fv (θv)

]≥ 1

|V|∑v∈V

EPv

[fv(θ)− fv (θv)

]

≥ 1− 1/q

d1/qδR

d∑j=1

P(

sign(θj

)6= −Vj

).

I As a result, the minimax error is lower bounded as

ε∗k (FG,2,Θ) ≥ 1− 1/q

d1/qδR

infv

d∑j=1

P(vj(Y 1, . . . , Y k

)6= Vj

) , (31)

where v denotes any testing function mapping Y tkt=1 to −1, 1d.

Proofs 60 / 73


I In the next, we lower bound the testing error by a total variation distance. To do so, we use

Le Cam’s inequality that for any set A and distributions P,Q, we have

P (A) +Q(Ac) ≥ 1− ‖P −Q‖TV .

I We split the coordinates into the positive parts and the negative parts.

P+j :=1

2d−1

∑v∈V:vj=1

Pv and P−j :=1

2d−1

∑v∈V:vj=−1

Pv.

I That is, P+j and P−j corresponds to conditional distributions over Y t given the events

vj = 1 and vj = −1.

Proofs 61 / 73


I Applying Le Cam’s inequality yields

P(vj(Y 1:k

)6= Vj

)=

1

2P+j

(vj(Y 1:k

)6= 1)

+1

2P−j

(vj(Y 1:k

)6= −1

)≥ 1

2

(1− ‖P+j − P−j‖TV

).

I Applying the Cauchy-Schwartz inequality , we have an upper bound for ‖P+j − P−j‖TV:

d∑j=1

‖P+j − P−j‖TV ≤√d

d∑j=1

‖P+j − P−j‖2TV

12

.

Proofs 62 / 73


I Then, we get a lower bound for the minimax error:

ε∗k (FG,2,Θ) ≥(

1− 1

q

)d1−1/qδR

2

1− 1√d

d∑j=1

‖P+j − P−j‖2TV

12

. (32)

I In the following, we present a sharp bound on∑dj=1 ‖P+j − P−j‖2TV.

Proofs 63 / 73

Proof of Proposition 1: The Third Part

I Defined the covariance matrix:

Σ := σ2

[‖θ‖22 〈θ, τ〉〈θ, τ〉 ‖τ‖22

]= σ2[θτ ]>[θτ ], (33)

with the corresponding shorthand Σt for the covariance computed for the tth pair (θt, τ t).

Lemma 3.

For each j ∈ 1, · · · , d, the total variation norm is bounded as

‖P+j − P−j‖2TV ≤ δ2k∑t=1

E

[ θtjτ tj

]> (Σt)−1

[θtjτ tj

] . (34)

Proofs 64 / 73


I Note the identity:d∑j=1

[θj

τj

][θj

τj

]>=

[‖θ‖22 〈θ, τ〉〈θ, τ〉 ‖τ‖22

]. (35)

I By Lemma 3, we have that

d∑j=1

‖P+j − P−j‖2TV ≤ δ2k∑t=1

E

d∑j=1

tr

(Σt)−1

[θtjτ tj

][θtjτ tj

]>=δ2

σ2

k∑t=1

E[tr((

Σt)−1

Σt)]

= 2kδ2

σ2.

Proofs 65 / 73


I By now, we find the nearly final lower bound:

ε∗k (FG,2,Θ) ≥(

1− 1

q

)d1−1/qδR

2

(1−

(2kδ2

dσ2

) 12

). (36)

I We now restrict (F, P ) ∈ FG,2 and we need to choose parameter σ2 and δ2 so that

E[||X||22

]≤ G2 for X ∈ N (δv, σ2Id).

I We show that the following parameters are sufficient:

σ2 =8G2

9dand δ2 =

G2

9min

1

k,

1

d

,

=⇒ 1−(

2kδ2

dσ2

) 12

≥ 1−(

18

72

) 12

=1

2and E

[‖X‖22

]=

8G2

9+G2d

9min

1

k,

1

d

≤ G2.

Proofs 66 / 73


I Plugging the chosen parameters into (36), we get the desired lower bound:

ε∗k (FG,2,Θ) ≥ 1

12

(1− 1

q

)d1−1/qRGmin

1√k,

1√d

=

1

12

(1− 1

q

)d1−1/qRG√

kmin1,

√k/d.

Proofs 67 / 73

Outline


Key Results

Smooth Optimization


Lower Bounds

Proofs

Proof of Theorem 1


Conclusion

Conclusion 68 / 73

Conclusion

I We focus on the stochastic, convex and zero-order optimization problems.

I Zero-order algorithms use two-point evaluations to approximate directional derivative that

the first-order methods utilize.

I For smooth optimization, we show that stochastic mirror descent based on zero-order

gradient estimate is only O(√

d)

slower than the one based on first-order information.

I For non-smooth optimization, we show that by the smoothing technique, its convergence

speed is at most O(√

log d)

worse than the one of smooth optimization.

I Lower bounds indicate that the proposed methods are minimax optimal.

Conclusion 69 / 73

References I

A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with

multi-point bandit feedback. In Proceedings of the 23rd Conference on Learning Theory,

pages 28–40, 2010.

P. L. Bartlett, V. Dani, T. P. Hayes, S. M. Kakade, A. Rakhlin, and A. Tewari. High-probability

regret bounds for bandit online linear optimization. In Proceedings of the 21st Conference on

Learning Theory, pages 335–342, 2008.

A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for

convex optimization. Operation Research Letters, 31(3):167–175, 2003.

J. C. Duchi, P. L. Bartlett, and M. J. Wainwright. Randomized smoothing for stochastic

optimization. SIAM Journal on Optimization, 22(2):674–701, 2012a.

Conclusion 70 / 73

References II

J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Finite sample convergence rates

of zero-order stochastic optimization methods. In Advances in Neural Information Processing

Systems 25, pages 1448–1456, 2012b.

J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order

convex optimization: The power of two function evaluations. IEEE Transaction on

Information Theory, 61(5):2788–2806, 2015.

A. Flaxman, A. T. Kalai, and H. B. McMahan. Online convex optimization in the bandit setting:

gradient descent without a gradient. In Proceedings of the 16th Annual ACM-SIAM

Symposium on Discrete Algorithms, pages 385–394, 2005.

K. G. Jamieson, R. D. Nowak, and B. Recht. Query complexity of derivative-free optimization.

In Advances in Neural Information Processing Systems 25, pages 2681–2689, 2012.

Conclusion 71 / 73

References III

C. Jin, L. T. Liu, R. Ge, and M. I. Jordan. On the local minima of the empirical risk. In

Advances in Neural Information Processing Systems 31, pages 4901–4910, 2018.

A. Nemirovski, A. B. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation

approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574–1609, 2009.

Y. Nesterov. Lectures on convex optimization. Springer, 2018.

Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions.

Foundations of Computational Mathematics, 17(2):527–566, 2017.

T. Salimans, J. Ho, X. Chen, and I. Sutskever. Evolution strategies as a scalable alternative to

reinforcement learning. arXiv, 1703.03864, 2017.

S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in

Machine Learning, 4(2):107–194, 2012.

Conclusion 72 / 73

References IV

O. Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In

The Proceedings of the 26th Annual Conference on Learning Theory, pages 3–24, 2013.

J. C. Spall. Introduction to stochastic search and optimization: estimation, simulation, and

control. John Wiley & Sons, 2005.

M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational

inference. Foundations and Trends in Machine Learning, 1(1-2):1–305, 2008.

D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural

evolution strategies. Journal of Machine Learning Research, 15(1):949–980, 2014.

Conclusion 73 / 73

zero-order convex optimization: an introduction

Documents