This work was done in collaboration with Thomas Foster, Jarek Liesen, Jamie Heagerty, Andreas Kirsch and Jakob Foerster. You can find the complete writeup including all definitions and proofs in my MSc thesis.

Code

Find the code for this project on my GitHub here and here. It contains the new scoring functions and new stochastic environments implemented in JAX for the Kinetix 2D physics-based environments.

Summary

To improve the robustness and generalisation of reinforcement learning (RL) agents, a common approach is to train on a large set of environments generated from an unsupervised environment design engine. If the engine can generate a wide variety of environments, it is not guaranteed that every level is useful for our training. Ideally, we want to train our agent on levels that are not trivial, not too difficult and relevant to the task we have in mind.

In this project, we propose a set of new scoring functions that aim to select levels with such characteristics at each point in time of the training run. The result is a curriculum of environments with increasing difficulty. Previous regret-based methods were shown to be outperformed by learnability-based methods. However, the learnability scoring function only applies to non-stochastic environments with binary rewards. We propose and try out new scoring functions that apply to stochastic environments with general reward scales. We also show that learnability is involved in an upper bound on the expected improvement of the agent.

The figure shows the performance of some of the scoring functions on randomly generated eval levels (left) and handmade eval levels (right).