Collaborative Research: HCC: Small: Modeling Learning Engagement Using AI and Gaze Data to Improve Virtual Reality Immersive Training

About This Grant

A strong U.S. workforce is vital to maintain a standard of excellence and global leadership in science, technology, engineering and mathematics (STEM). However, the pace in which Americans are entering the STEM workforce is too slow to meet the pressing needs of industries like healthcare, military, manufacturing, and construction. New training methods are needed to improve how learners acquire STEM knowledge and skills to enter the U.S. STEM workforce. Immersive virtual reality (VR) as a training platform is a solution that can provide individual training and education for all Americans. VR experiences with high engagement are particularly effective as mechanisms for learning new and complex ideas. However, without real-time instructor feedback, many learners struggle to stay engaged. This project explores how eye tracking and artificial intelligence (AI) can be used to measure and guide learner engagement in VR training. By automatically detecting when learners are focused or distracted, the system can provide visual cues to support attention and understanding. The outcomes of this work will help more Americans gain the knowledge and skills needed for high-demand STEM careers. Also, novel opportunities will be opened for improving training approaches for people with attention or cognitive challenges. Overall, the knowledge generated from this project will enhance STEM training outcomes when using VR. This work will empower more Americans to advance our national welfare, prosperity and security. This project investigates how gaze-based models can be used for estimating engagement and reactively guiding user attention within immersive VR training experiences. Specifically, VR training will take place within construction safety training, as this domain has a rapidly growing workforce similar to other STEM disciplines. Currently, the speed at which construction workers and professionals gain knowledge and skills needed to complete job tasks safely remains inadequate. To perform construction work safely, trainees are required to be highly engaged during short, demanding, complex training interventions to reduce fatalities and injuries in the workplace. The project is structured around three thrusts in the following areas: (1) data collection and modeling approaches, (2) reactive designs to support engagement; and (3) validation with construction professionals. First, the project will develop real-time and post annotation methods to label engagement states. This process will use gaze, engagement, and training performance data from VR training sessions. These data will be used to reduce labeling bias and improve model accuracy. Then, machine learning techniques (e.g., deep-learning, spike, liquid networks) will be used to identify key fixations, saccades, and pupillary features as they relate to engagement and performance for building real-time engagement models. Various time scales, features, and models will be explored to produce optimal real-time estimations of learner engagement. Second, the research team will investigate how different activation functions can be used to trigger visual attention cues in response to user engagement levels (low, medium, high). Human-centered interventions will assess how these functions influence learner ability to initiate, sustain, or regain engagement. Third, the project will study the produced models using VR construction safety training sessions with industry professionals. The effectiveness and transferability of real-time attention guidance will be evaluated to improve VR trainings. Ultimately, these thrusts will advance our scientific understanding of gaze-based engagement modeling to learn construction safety materials, use AI to guide visual attention, and improve VR tools for workforce training outcomes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.