CAREER: Networked Multi-User Augmented Reality for Mobile Devices

About This Grant

Augmented reality (AR) is an emerging technology with potential applications in many fields, including education, entertainment, and public safety. In single-user AR, a user's perception is augmented by overlaying virtual objects onto the real world. In multi-user AR, multiple users are able to view and interact with a common set of virtual objects. However, in today's multi-user AR applications, users can experience poor performance (such as high latency or inconsistent views of the virtual objects by different users) due to the AR application's lack of advanced networking capabilities. This project seeks to address such user experience issues by developing the network capabilities of the underlying AR platform, in order to support future multi-user AR applications. The vision of this project is to create AR-enabling technology where multiple users can visually scan the environment, and a common set of relevant virtual objects swiftly pop up on top of the real-world objects. To accomplish this, the AR devices need to efficiently compute and exchange information amongst themselves, in order to synchronize their joint understanding of the real and virtual worlds. The planned research has three main technical components: (1) measuring and characterizing multi-user AR applications; (2) adapting an AR application to the computing and networking conditions; and (3) adapting the network to the needs of an AR application. Evaluation scenarios include an educational AR classroom application, as well as mobile clients in a 5G testbed. The success of this project will open up a new class of multi-user AR applications in a broad range of fields, including education (for example, multiple AR-equipped students in a classroom) and public safety (for example, multiple AR-equipped first responders in a disaster scenario). The findings from this project will impact how application developers design AR applications and how network operators manage AR network traffic. The multi-user AR platform will be incorporated into undergraduate classes and a campus makerspace, as well as in outreach efforts at local hackathons and high school workshops, with the aim of encouraging women to enter and stay in engineering fields. Results from this project, including papers, simulation and application code, and AR quality evaluation and benchmarking tools, will be available at the project website https://github.com/multi-user-ar. The website and code will be available and supported for the duration of this project, and preserved online thereafter. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.