A Multiscale Mixed-Graph Neural Network Based on Kinematic and Dynamic Joint Features for Human Motion Prediction

Predicting human future motion holds significant importance in the domains of autonomous driving and public safety. Kinematic features, including joint coordinates and velocity, are commonly employed in skeleton-based human motion prediction. Nevertheless, most existing approaches neglect the critical role of dynamic information and tend to degrade as the prediction length increases. To address the related constraints due to single-scale and fixed-joint topological relationships, this study proposes a novel method that incorporates joint torques estimated via Lagrangian equations as dynamic features of the human body. Specifically, the human skeleton is modeled as a multi-rigid body system, with generalized joint torques calculated based on the Lagrangian formula. Furthermore, to extract both kinematic and dynamic joint information effectively for predicting long-term human motion, we propose a Multiscale Mixed-Graph Neural Network (MS-MGNN). MS-MGNN can extract kinematic and dynamic joint features across three distinct scales: joints, limbs, and body parts. The extraction of joint features at each scale is facilitated by a single-scale mixed-graph convolution module. And to effectively integrate the extracted kinematic and dynamic features, a KD-fused Graph-GRU (Kinematic and Dynamics Fused Graph Gate Recurrent Unit) predictor is designed to fuse them. Finally, the proposed method exhibits superior motion prediction capabilities across multiple motions. In motion prediction experiments on the Human3.6 dataset, it outperforms existing approaches by decreasing the average prediction error by 9.1%, 12.2%, and 10.9% at 160 ms, 320 ms, and 400 ms for short-term prediction and 7.1% at 560 ms for long-term prediction.