Sunday 09 March 2025
Deep learning has become an essential tool in medical imaging, allowing researchers to develop sophisticated algorithms that can analyze and diagnose a wide range of conditions. However, one major challenge in this field is dealing with variability in input data – patients often have different numbers of modalities (e.g., MRI, CT scans) and the quality of these images can vary significantly.
To address this issue, researchers have turned to self-supervised pre-training methods, which involve training a model on a large dataset without explicit labels. This approach has shown promise in improving performance on downstream tasks, but it typically requires a fixed number of input modalities.
A new paper published in IEEE Transactions on Medical Imaging tackles this problem by introducing an adaptive masked autoencoder (AMAE) framework that can handle variable numbers of input modalities during both training and testing. The AMAE approach uses a dynamic convolutional layer to extract modality-specific features from each 3D image, which are then fed into a transformer encoder.
The authors tested their approach using a dataset of 45,364 MRI scans with varying numbers of contrasts (e.g., T1-weighted, T2-weighted). They found that the AMAE pre-trained weights enhanced the performance of both a finetune architecture and an original UNETR model on the task of infarct segmentation.
One notable observation from the results is that generating patches from all input images together leads to slightly better performance in the fine-tuning task. This suggests that the AMAE framework can effectively capture relationships between different modalities, even when they are not explicitly labeled.
The authors also evaluated their approach on a subset of 215 subjects where one contrast (AxT2) was absent. The results showed that the AMAE pre-trained weights with the original UNETR decoder achieved a mean Dice score of 0.575, which is only 2.9% lower than its performance when all contrasts were present.
Overall, this research demonstrates the potential of adaptive masked autoencoders in handling variable input modalities and improving performance on downstream tasks. As medical imaging continues to play an increasingly important role in healthcare, developing more flexible and robust algorithms like AMAE will be essential for unlocking its full potential.
The authors’ approach has several limitations, including the use of a fixed masking ratio during pre-training and the assumption that all modalities are equally informative. Future research could explore varying masking ratios and incorporating domain-specific knowledge to improve performance on specific tasks.
Cite this article: “Adaptive Masked Autoencoders for Handling Variable Input Modalities in Medical Imaging”, The Science Archive, 2025.
Medical Imaging, Deep Learning, Adaptive Masked Autoencoder, Self-Supervised Pre-Training, Mri Scans, Ct Scans, Variable Input Modalities, Infarct Segmentation, Unetr Model, Image Segmentation
