Using Deep Multimodal Clustering to Characterize Motor Phenotypes Across the Lifespan

About This Grant

PROJECT SUMMARY/ABSTRACT Traditional clinical measures to assess motor impairments typically focus on the overall outcome of the movement, rather than examining the specific kinematics underlying how the movement was performed. Kinematic based analyses are likely to be a significant improvement over traditional measures, as they can be more sensitive to mild impairments or small amounts of change, and can detect specific impairments including compensatory strategies to guide targeted interventions. However, there are many challenges in applying kinematic analysis to the upper limbs, including the need for specialized equipment and the lack of metrics that can reflect the rich spatiotemporal movement patterns seen in arm motion. Even within an unimpaired population, there is significant heterogeneity in the movement strategies individuals employ, and these strategies may change across the lifespan. Our long-term goal is to infer a patient’s “movement phenotype” from a recording of their movement that can be done in the clinic with limited equipment and time. As a first step, the objective of this 3-year project is to develop the methodology to identify motor phenotypes in a normative population using multimodal sensing and deep learning-based clustering strategies. We will first collect a large normative dataset on 400 individuals (ages 5-85+) using multiple camera views and inertial measurement units (IMUs) while participants perform prescribed unilateral and bilateral reaching movements and a functional bilateral pouring and drinking task. We will then use deep representation networks to encode the motion in latent space and apply optimization-based clustering over these latent encodings to determine movement phenotypes. After identifying motor phenotypes from all tasks on the full multimodal dataset (aim 1), we will then determine a clinically optimal subset of sensors and tasks required for reliable phenotyping (aim 2) via algorithmic means including phenotype-aware knowledge distillation. These reduced, clinic-friendly configurations will be evaluated against the full system for cluster fidelity, robustness to noise, and practical feasibility. This work advances the state-of-the-art by fusing markerless vision and wearable IMUs with self- supervised contrastive representation learning and phenotype-aware knowledge distillation to capture normative motor phenotypes in a clinically viable manner. At the conclusion of this three-year project, we will have both a multimodal normative dataset and methodology that we can then apply to clinical populations across the lifespan. We expect that the ability to phenotype movements in clinical populations will represent a large step towards precision rehabilitation.