Friday 28 February 2025
A team of researchers has made significant strides in developing a system that can automatically translate comic books and manga from Indonesian to English, opening up new possibilities for global audiences.
The project aimed to address the challenges posed by low-resource languages like Indonesian, which have limited datasets and linguistic complexities. By combining computer vision, optical character recognition (OCR), and machine translation techniques, the team created a pipeline that can efficiently detect and extract text from comic book panels, transcribe it accurately, and then translate it into English.
The system’s first task was to identify speech bubbles within the comic book images using YOLOv5xu, a lightweight object detection model. This fine-tuned model demonstrated impressive recall and precision, successfully detecting even small speech bubbles in complex layouts.
Next, the pipeline employed Tesseract OCR to transcribe the extracted text from the speech bubbles. Despite challenges with stylized fonts and noisy backgrounds, the system achieved strong character-level recognition accuracy, with an average character error rate of 3.1% and a word error rate of 8.6%.
The final step involved translating the transcribed text into English using MarianMT, a neural machine translation model fine-tuned on OpenSubtitles and Identic datasets. This approach allowed for effective handling of conversational text and idiomatic expressions, resulting in high semantic retention and formal translation quality.
The system’s performance was evaluated using metrics such as mean precision, mean recall, F1 score, character error rate, word error rate, BLEU score, and Meteor score. The results demonstrated the pipeline’s strong performance across all components, with the YOLOv5xu model achieving an mAP@0.5 of 88.9%, Tesseract OCR delivering a CER of 3.1% and a WER of 8.6%, and MarianMT scoring a BLEU score of 27% and a Meteor score of 61%.
This achievement has significant implications for the global accessibility of comic books and manga, as well as the potential to apply similar techniques to other image-based content across underrepresented language pairs. The system’s scalability and adaptability make it an attractive solution for publishers, translators, and enthusiasts alike.
The research highlights the importance of domain-specific datasets, effective pre-processing, and targeted fine-tuning in achieving high-performance machine translation results.
Cite this article: “Automated Comic Book Translation: A Breakthrough for Global Accessibility”, The Science Archive, 2025.
Comic Books, Manga, Machine Translation, Indonesian, English, Computer Vision, Ocr, Object Detection, Neural Network, Natural Language Processing
