CAREER: Modeling Situated Intention during Nondeterministic Pedestrian-Vehicle Interactions through Explainable Compositional Learning of Naturalistic Driving Data

About This Grant

With the fast progress of artificial intelligence, vehicles with higher levels of automation are entering daily life. Automated driving technologies are expected to improve traffic safety, promote travel efficiency, protect the environment, reduce mobility barriers for older generations or people with disabilities, and thus deliver overall societal benefits. However, existing algorithms embedded in autonomous vehicles still face fundamental challenges in recognizing the quickly changing intentions of pedestrians moving on the road and sidewalks, making it hard to predict their behavior and plan vehicle motions. Such limitations impede the implementation of fully autonomous and safe cars in city environments and create additional risks for pedestrians and other road users. This project focuses on developing novel techniques to model and predict the complex and changing intentions of pedestrians. By learning the thinking process of drivers and their driving responses, an algorithm will be created to equip the automated cars with similar capabilities to interact with pedestrians and other road users smoothly and safely. The research process will include naturalistic driving data collection, subject experiments, knowledge modeling, and learning algorithm development. The developed algorithm will be evaluated in an immersive virtual environment. The project also includes activities to promote user-centered design in engineering education, foster the awareness of biases and ethical issues related to artificial intelligence technologies, and increase the participation of underrepresented communities in Science, Technology, Engineering, and Mathematics. This project surmounts limitations of current pedestrian behavior prediction to achieving mutual intelligibility between autonomous vehicles and pedestrians. Principally, unlike the traditional static view of pedestrian intention at a critical moment, this research investigates the relationship between non-verbal actions and intention changes of pedestrians moment-to-moment in dynamic (changing) and interactive situations. The project collects temporal video segments and human reasoning descriptions simultaneously through event-segmentation-based video experiments. Then, it develops a compositional learning method to learn and combine language features with visual features. This method can avoid the rigid structure of expert-selected feature space by creating collective features from ordinary drivers, and the three-level explainability of the learning model can justify model outputs from input features. Finally, the developed intention prediction model will be evaluated through subject experiments in a virtual interactive pedestrian simulator. The research findings will be shared through industrial collaborators and conferences, and a pedestrian behavior benchmark dataset will be disseminated to the public. The research results will be included in engineering education to promote design approaches that take into account the users’ feelings, values, and overall mental state (empathic design). These educational activities will include courses at the investigator’s university, the autonomous driving research community, and industry. They will increase the awareness of critical human-centered AI issues like biases, trust, and social intelligence in design of AI. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.