Friday 14 March 2025
A recent study has shed light on the biases inherent in large language models (LLMs), specifically how they perform when predicting outcomes based on socio-demographic factors such as gender, age, education level, and political ideology. The research highlights the disparities in accuracy across different demographic groups, underscoring the need for more inclusive and fair AI systems.
The investigation employed a dataset comprising survey responses from Chile and the United States, with participants asked to answer questions related to presidential voting, abortion, and constitutional referendums. Four models, including ChatGPT and Llama-13B, were fine-tuned on these datasets and evaluated based on their predictive accuracy.
Results showed that all models demonstrated varying levels of bias against certain demographic groups. In Chile, women, older adults, and individuals with lower education levels received less accurate predictions compared to men, younger adults, and those with higher education. Conversely, in the United States, men, individuals from metropolitan regions, and those with higher education exhibited better performance.
The study also found that models relied heavily on political variables to achieve strong predictive performance. In Chile, where politics are less polarized, removing these variables did not significantly impact accuracy. However, in the United States, where political divisions are more pronounced, omitting political factors resulted in a significant decrease in performance.
Furthermore, the research analyzed the sensitivity of models to prompt variations. The results indicated that different models responded differently to changes in language structure and context. Some models performed better when using Spanish prompts or omitting examples, while others showed improved accuracy with fewer shot examples.
The findings have important implications for AI development and deployment. They suggest that LLMs require careful consideration of socio-demographic factors to ensure fair and inclusive performance. Additionally, the study highlights the need for more nuanced approaches to prompt design and fine-tuning, as different models respond differently to varying input conditions.
To address these biases, researchers recommend exploring novel methods for incorporating diverse perspectives and experiences into AI development. This may involve incorporating more diverse training data, using techniques such as data augmentation or adversarial training to improve robustness, or developing new algorithms that explicitly account for socio-demographic factors.
As the use of LLMs continues to grow in various applications, including education, healthcare, and social sciences, it is essential to ensure that these systems are not perpetuating harmful biases.
Cite this article: “Large Language Models Biases Exposed: A Call for Fairer AI Systems”, The Science Archive, 2025.
Large Language Models, Bias, Socio-Demographic Factors, Predictive Accuracy, Demographic Groups, Chatgpt, Llama-13B, Political Variables, Prompt Variations, Ai Development
Reference: Andrés Abeliuk, Vanessa Gaete, Naim Bro, “Fairness in LLM-Generated Surveys” (2025).
