Friday 21 March 2025
A team of researchers has made significant progress in developing a multilingual named entity recognition (NER) model that can accurately identify and classify entities such as people, locations, organizations, and dates across multiple Indian languages. The model, which combines data from three existing datasets, has achieved an F1 score of 92.11%, outperforming previous models in the field.
Named entity recognition is a crucial task in natural language processing (NLP), with applications in areas such as information extraction, text summarization, and question answering. However, the lack of standardization and inconsistencies in tagging across different languages have made it challenging to develop effective NER models for Indian languages.
The researchers tackled this problem by combining data from three existing datasets: HiNER, MahaNER, and B-NER. They mapped the entity classes from these datasets into a consistent schema, reducing inconsistencies and increasing the overall quality of the data. The resulting dataset consists of over 100,000 manually annotated sentences across Hindi, Bengali, and Marathi.
The researchers trained two custom models: a transformer-based model and a fine-tuned version of XLM-RoBERTa (XLM-R). They also experimented with pre-trained multilingual models such as mBERT and MuRIL. The results showed that the fine-tuned XLM-R model outperformed all other models, achieving an F1 score of 92.11%.
The model’s performance was evaluated using a variety of metrics, including precision, recall, and F1 score. The results indicate that the model is particularly effective in recognizing entities such as people, locations, and organizations, with high precision rates across all three languages.
The development of this multilingual NER model has significant implications for applications such as text summarization, question answering, and information extraction. For example, the model could be used to automatically extract relevant information from news articles or social media posts in multiple Indian languages.
In addition to its practical applications, this research contributes to a deeper understanding of the challenges and opportunities in developing NER models for low-resource languages. The findings highlight the importance of standardization and consistency in tagging across different languages, as well as the potential benefits of combining data from multiple sources.
The researchers’ approach has set a new benchmark for multilingual NER in Indian languages, and their model has the potential to be used in a variety of applications.
Cite this article: “Multilingual Named Entity Recognition Model Achieves High Accuracy Across Indian Languages”, The Science Archive, 2025.
Named Entity Recognition, Multilingual, Indian Languages, Natural Language Processing, Nlp, Text Summarization, Question Answering, Information Extraction, Xlm-R, Mbert
