Thursday 06 March 2025
A new study has shed light on how large language models (LLMs) process and represent linguistic concepts across different languages. Researchers found that LLMs, which are trained on vast amounts of text data, often share similar representations of grammatical concepts such as gender, number, and tense across multiple languages.
The study analyzed two popular LLMs, Llama-3-8B and Aya-23-8B, which were trained on large datasets of text. The researchers used a technique called sparse autoencoders to identify the features that are most important for representing these grammatical concepts. They found that many of these features are shared across multiple languages, even if they are not closely related.
For example, the study found that the feature associated with masculine gender is shared across 11 languages in Llama-3-8B, while the feature associated with past tense is shared across 13 languages. Similarly, the feature associated with accusative case is shared across 10 languages. These findings suggest that LLMs have developed a common representation of these grammatical concepts that can be applied to multiple languages.
The researchers also found that the number of languages that share a given feature follows a power-law distribution, meaning that most features are shared across only a few languages, while a smaller number of features are shared across many languages. This suggests that LLMs have developed a hierarchical representation of linguistic concepts, with more general features being shared across multiple languages and more specific features being unique to individual languages.
The study’s findings have implications for the development of machine translation systems, which rely on LLMs to translate text from one language to another. By identifying the common features that are used to represent grammatical concepts across languages, researchers can develop more accurate and efficient machine translation systems that can handle multiple languages.
The researchers also explored the use of counterfactual datasets to test the robustness of LLMs to linguistic variations. Counterfactual datasets consist of pairs of sentences that differ in a specific way, such as changing the subject-verb agreement or the tense of a verb. By training LLMs on these datasets, the researchers found that they can improve their ability to translate text from one language to another.
The study’s findings also have implications for natural language processing (NLP) more broadly. By understanding how LLMs process and represent linguistic concepts across languages, researchers can develop more accurate and efficient NLP systems that can handle multiple languages.
Cite this article: “Common Representations of Grammatical Concepts in Large Language Models Across Languages”, The Science Archive, 2025.
Large Language Models, Linguistic Concepts, Grammatical Concepts, Gender, Number, Tense, Machine Translation Systems, Counterfactual Datasets, Natural Language Processing, Nlp, Sparse Autoencoders
