Friday 14 March 2025
The quest for a more intelligent virtual assistant has long been a holy grail for tech enthusiasts and researchers alike. The latest development in this pursuit is a modular, retrieval-augmented generation (RAG) system designed to answer questions about the University of S˜ao Paulo. This innovative architecture combines the strengths of both retrieval-based and generative models to provide more accurate and informative responses.
At its core, the RAG system consists of two primary components: a retriever and a generator. The former is responsible for identifying relevant text chunks from a database of official documents, while the latter uses these retrieved texts to generate answers to user queries. By leveraging both approaches, the system can capitalize on the strengths of each, producing more comprehensive and accurate responses.
To evaluate the effectiveness of this RAG-based virtual assistant, researchers compiled a corpus of official documents related to the University of S˜ao Paulo and created a question-answer dataset. They then conducted extensive experiments with various retrieval and generative models, exploring different hyperparameters such as chunk size and the number of chunks provided.
The results indicate that the system’s performance is significantly hampered by retrieval failures, primarily due to limitations in multilingual embedding models. However, when the correct document chunks are supplied to the language generation model, accuracy improves dramatically. In fact, the optimal retriever model achieves a Top-5 accuracy of 30%, while the most effective generative model scores 22.04% against ground truth answers.
The researchers also conducted a qualitative evaluation by manually assessing the generated responses. While there were some issues with the system’s ability to handle complex queries and provide accurate information, overall, the results suggest that this RAG-based virtual assistant has tremendous potential for improving question-answering capabilities.
One of the key benefits of this approach is its flexibility and adaptability. By incorporating a wide range of sources and datasets, the system can be trained to answer questions on a vast array of topics and domains. Additionally, the modular architecture allows researchers to easily experiment with different models and hyperparameters, fine-tuning the system for specific use cases.
As the quest for more intelligent virtual assistants continues, this RAG-based system represents an important step forward in the development of more accurate and informative language understanding capabilities. By leveraging the strengths of both retrieval-based and generative models, researchers can create systems that are better equipped to handle complex queries and provide users with more comprehensive and accurate information.
Cite this article: “Advancing Virtual Assistants: A Modular Retrieval-Augmented Generation System”, The Science Archive, 2025.
Virtual Assistant, Retrieval-Augmented Generation, University Of S˜Ao Paulo, Modular Architecture, Language Understanding, Question-Answering, Multilingual Embedding Models, Chunk Size, Generative Models, Ground Truth Answers.
