CAREER: Unsupervised and Autonomous Reinforcement Learning of Skills

About This Grant

This project will study how artificial intelligence (AI) models can learn new tasks. Today, AI models are often taught to perform a task (e.g., controlling a self-driving car) by showing them examples of what a human would do. This project builds upon an area of research known as reinforcement learning, where AI models learn by trial and error, much like a dog might learn a trick by trying different behaviors and receiving a treat for the correct one. A major challenge in existing algorithms for trial-and-error learning is that complex tasks, such as assembling a house with a robotic arm, may require hundreds of small steps to complete. If the AI model receives feedback (a success or failure signal) only after completing the entire task, then it is difficult to figure out what went wrong in all the small intermediate steps. This project takes three key steps to address this challenge. First, the research will develop new algorithms to discover small, reusable skills. For a robotic arm, instead of teaching it how to build an entire house at once, the AI model might first learn a skill for stacking blocks, then another for sorting blocks, and so on. Importantly, the AI model discovers these skills by exploring and experimenting, without requiring human demonstrations or hand-written code. Once learned, these skills can be rapidly combined and adapted to solve new, more complex tasks. Second, the research will create new simulators and algorithms that leverage GPUs to significantly decrease the time required to learn a new task; tasks that previously took hours of computation time will now require just a few minutes. Third, the research will provide mathematical explanations that show when and why learning these skills will be effective. Taken together, this research will enable new algorithms for efficiently and robustly solving decision-making problems, with potential applications ranging from safely controlling self-driving cars to efficiently controlling factories. Reinforcement learning (RL) has the potential to address many of the challenges in machine learning and AI today, enabling AI systems to reason about the consequences of their actions and to optimize their actions for long-term outcomes. In RL, an agent interacts with an environment, learning how to maximize a reward function through trial and error, often discovering strategies that are better and more robust than those designed by human experts. However, practical problems hinder the adoption of RL systems today: designing and implementing reward functions is difficult, and current RL algorithms require a prohibitively large amount of computational power. This research will leverage connections between self-supervised learning and RL to address these challenges, opening the door to new users and applications of RL. The first aim of this research is to develop a unified theory of existing methods in this space, starting from the observation that self-supervised learning, generative AI, and RL can all be defined in terms of information-theoretic and probabilistic quantities. The second aim is to build a new skill-learning algorithm that learns an exponentially large repertoire of skills by leveraging a novel hierarchical representation. This hierarchical representation will also enable new exploration strategies that utilize GPU-accelerated simulation. The third aim of the research is to use this repertoire of skills to develop new ways for users to interact with RL systems, to expand the planning horizon of RL methods, and to improve the exploration and robustness of RL agents. Because skills and their associated representations are learned with information-theoretic objectives, tools from probability theory can be used to analyze how best to use skills for these downstream applications. In parallel, this research will develop a curriculum of Jupyter notebooks for teaching RL to learners across the United States, helping to prepare the next generation of scientists and a future-ready AI workforce. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.