CIRC: Infrastructure for Large-Scale Procedural Generation of Photorealistic 3D Data

About This Grant

Data is critical to the advances of AI, especially computer vision. Large-scale labeled data, in particular, has been a critical driver of progress in computer vision, the task of making computers see like humans. At the same time, data has also been a major challenge, as many important vision tasks remain starved of high-quality data. This is especially true for 3D vision, which seeks to make computers see 3D shapes from 2D images. To teach computers to see 3D shapes, many images annotated with the 3D shapes are needed, but such annotated images are difficult to obtain in the real world. This project proposes to address the data challenge by using synthetic images, images created by a computer instead of being captured from the real world. In particular, the project proposes to build Infinigen, a free and open-source software program that uses mathematical rules and physics simulation to create realistic-looking 3D scenes covering a wide range of objects in the real world. Infinigen can create an infinite number of annotated images that look like real images and are useful for training AI systems. Compared to existing sources of synthetic data, Infinigen is unique for a number of reasons. It is not a finite collection of 3D assets or synthetic images; instead, it is a generator that can create infinitely many distinct shapes, textures, materials, and scene compositions. Every asset, from shape to texture, from macro structures to micro details, is entirely procedural, generated from scratch via randomized mathematical rules that allow infinite variation and composition. When completed, Infinigen will offer a broad coverage of objects and scenes in the real world from natural objects to artificial ones, including plants, animals, terrains, cities, indoors, and natural phenomena such as fire, cloud, rain, and snow. This project is awarded as a planning project. The project team will take initial steps to engage the community and make preparations for developing Infinigen into large-scale community infrastructure. The project team will build upon an existing prototype of Infinigen and make improvements in terms of capabilities and user services. Infinigen is expected to benefit many CISE disciplines including computer vision, computer graphics, machine learning, and robotics. It can create training data for a wide range of computer vision tasks, including object detection, semantic segmentation, pose estimation, 3D reconstruction, view synthesis, and video generation. Work in this project will be integrated with K12, undergraduate, and graduate education through research training, course development and outreach events. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.