Friday 21 March 2025
A new approach to human mesh recovery has been proposed, one that combines vision-language models and image observations to produce accurate 3D poses from a single image. The technique, developed by researchers at Sichuan University, uses a diffusion-based framework to refine initial predictions guided by gradients derived from both 2D observations and text descriptions.
The problem of human mesh recovery is a challenging one, as it requires accurately reconstructing the shape and pose of a person’s body from a single image. Current methods often struggle with aligning 3D models to 2D images, leading to suboptimal results. To overcome this limitation, the researchers turned to vision-language models (VLMs), which have shown impressive ability in tasks such as text-to-image synthesis and visual question answering.
The approach begins by training a text encoder and a pose VQ-VAE (Variational Autoencoder) to align texts with body poses in a shared latent space. This allows the model to incorporate prior knowledge from VLMs into its predictions. Next, the researchers employ a diffusion-based framework to refine initial predictions guided by gradients derived from both 2D observations and text descriptions.
The key innovation here is the use of adaptive normalization layers in the diffusion process. These layers combine noisy SMPL (Simple Pose and Line Model) parameters with positional embeddings and observations through adaptive normalization. This allows the model to effectively balance image observations and model assumptions, leading to more accurate predictions.
To evaluate their approach, the researchers tested it on two popular benchmarks: 3DPW (3D Pose in the Wild) and Human3.6M. The results were impressive, with their method outperforming existing state-of-the-art methods by a significant margin. For example, on the 3DPW dataset, their method achieved an MPJPE (Mean Per-Joint Position Error) of 69.3mm, compared to 87.8mm for the standard Gaussian distribution.
The researchers also provided more qualitative results, showcasing the ability of their approach to produce accurate poses from a single image. The images demonstrate good alignment and 3D accuracy, outperforming existing methods such as HMR2.0 and ProHMR.
This new approach has significant implications for applications such as 3D pose estimation, human-computer interaction, and robotics. By leveraging the strengths of vision-language models and diffusion processes, it offers a powerful tool for recovering accurate 3D poses from single images.
Cite this article: “Accurate Human Mesh Recovery Using Vision-Language Models and Diffusion Processes”, The Science Archive, 2025.
Human Mesh Recovery, Vision-Language Models, Image Observations, Diffusion-Based Framework, Pose Estimation, 3D Poses, Single Images, Adaptive Normalization Layers, Smpl Parameters, Text Encoder
