Thursday 26 June 2025
As language models continue to advance, researchers are uncovering new insights into how they acquire and transfer cultural knowledge across languages. A recent study has shed light on this phenomenon, revealing that large language models (LLMs) may exhibit different patterns of cross-lingual transfer depending on the type of culture being studied.
The investigation focused on four non-Anglophonic cultures: Koreans, Han Chinese, Tibetans, and Mongols in China. The researchers used a combination of pre-existing datasets and newly constructed probing questions to assess the LLMs’ knowledge of each culture’s customs, history, and traditions.
One key finding was that when continually pretrained on English language data, LLMs tend to exhibit bidirectional cultural transfer between high-resource languages such as English and Korean. This means that the models can easily learn about each other’s cultures and transfer this knowledge back and forth.
However, for low-resource languages like Tibetan and Mongolian, the picture is more complex. The study found that these languages primarily transfer knowledge to English with limited reverse flow. This suggests that LLMs may be more inclined to absorb cultural information from high-resource languages, rather than vice versa.
So what drives this asymmetric phenomenon? According to the researchers, it may come down to frequency of appearance in pretraining data. Cultural knowledge that appears frequently in an LLM’s training corpus is more likely to be transferred across languages, regardless of the culture being studied.
To test this hypothesis, the team constructed probing questions for each culture and evaluated the models’ performance using a cloze-style format. The results supported their theory, with cultural knowledge appearing more frequently in English pretraining data transferring more easily to other languages.
The study also explored how LLMs acquire Anglophonic cultural knowledge during English pretraining. By analyzing the attributes of famous individuals from core Anglosphere countries, the researchers found that this knowledge can indeed be transferred to non-English languages after continual pretraining.
This research has significant implications for the development of culturally aware language models. As we strive to create machines that can understand and interact with diverse cultures, it’s crucial to consider the role of cultural transfer in shaping their knowledge and biases.
The findings also highlight the importance of considering the frequency of appearance in pretraining data when evaluating an LLM’s ability to acquire cultural knowledge. By better understanding these dynamics, researchers can develop more effective strategies for cultivating culturally aware language models that truly serve the needs of diverse communities.
Cite this article: “Unpacking Cross-Lingual Cultural Transfer in Large Language Models”, The Science Archive, 2025.
Large Language Models, Cross-Lingual Transfer, Cultural Knowledge, Non-Anglophonic Cultures, Pretraining Data, Frequency Of Appearance, Anglophonic Culture, Cloze-Style Format, Probing Questions, Cultural Awareness
