Abstract: A core ingredient of general artificial intelligence is the ability of AI to improve itself, including its own learning algorithm.In recent years meta-learning has made large progress in producing models that can learn very quickly from a few examples using in-context learning, fast weights, and optimization-based methods. This tremendously helps in adapting to new, similar, problems, but it does not automate ML Research itself. In this talk, I discuss meta-learning general-purpose learning algorithms. I present different approaches to meta-generalization from learned loss functions, weight-shared LSTMs that implement gradient descent in their recurrent dynamics, to black-box Transformers that learn how to learn generally. We discover phase transitions where models suddenly transition from memorization to task identification, to general learning to learn. We identify crucial ingredients such as memory capacity and practical interventions in meta-training.

Bio: Louis is a final-year PhD student at the Swiss AI Lab IDSIA, advised by Prof. Jürgen Schmidhuber. He received his MRes in Computational Statistics and Machine Learning from University College London (1st rank) and interned at Google and DeepMind. His research focus is on meta-learning general-purpose learning algorithms. This describes learning algorithms that generalize across significantly different tasks and environments, towards automating ML research itself. In his works MetaGenRL, VSML, SymLA, GPICL, and others he investigates how to meta-learn RL algorithms, discover novel general learning algorithms that do not rely on gradient descent, general-purpose in-context learning, and self-referential meta-learning without meta optimization. Louis has co-organized several workshops at ICLR and has been an invited speaker for workshops at NeurIPS and ICML. Further, he won several GPU compute awards for the Swiss national supercomputer.