Wednesday 22 January 2025
The quest for better translation systems has been a longstanding challenge in the field of artificial intelligence. Recently, researchers have made significant progress by leveraging large language models (LLMs) to improve machine translation. However, this advancement is not without its limitations. In particular, LLMs struggle with figurative expressions, such as idioms and proverbs, which are common in many languages.
To address this issue, a team of researchers has developed a new dataset called the Proverb in Conversation Translation (PCT) dataset. This dataset consists of four language pairs – English to German, Indonesian, Mandarin Chinese, and Bengali – and contains over 1,000 proverbs with their corresponding translations. The dataset is unique because it includes contextual information, such as dialogue and conversation, which helps LLMs better understand the nuances of each proverb.
The researchers tested several LLMs on the PCT dataset using various prompts and evaluation metrics. They found that the best-performing model was a 7B-parameter LLM, which achieved a BLEU score of 42.8 and a CHRF++ score of 34.6. These results are significant because they demonstrate that LLMs can improve translation quality even when faced with figurative expressions.
The researchers also analyzed the performance of different prompts and evaluation metrics on the PCT dataset. They found that the best prompt was a contextualization-through-concatenation prompt, which involves concatenating the source text with its context to help the LLM better understand the proverb. This prompt achieved a BLEU score of 41.4 and a CHRF++ score of 39.7.
The study’s findings have important implications for the development of machine translation systems. They suggest that incorporating contextual information and using the right prompts can significantly improve the performance of LLMs on figurative expressions. The PCT dataset is also an important resource for researchers, providing a new benchmark for evaluating the capabilities of LLMs.
In addition to its technical significance, this study highlights the importance of cultural understanding in machine translation. Proverbs and idioms are often deeply rooted in culture and can be difficult to translate accurately. The PCT dataset and the research it has enabled demonstrate the need for more nuanced approaches to machine translation that take into account the complexities of human language.
Overall, the development of the PCT dataset and the testing of LLMs on this dataset represent an important step forward in the quest for better machine translation systems.
Cite this article: “Improving Machine Translation with Contextualized Proverbs”, The Science Archive, 2025.
Machine Translation, Large Language Models, Proverbs, Idioms, Cultural Understanding, Artificial Intelligence, Natural Language Processing, Llms, Bleu Score, Chrf++ Score
