Wednesday 16 April 2025
For centuries, grading student answers has been a time-consuming and labor-intensive task for educators. With the rapid advancement of artificial intelligence (AI), researchers have been exploring ways to automate this process using large language models (LLMs). A recent study published in the Journal of LaTeX Class Files presents an innovative approach to automated short answer grading, aptly named Grade Guard.
The traditional method of grading short answers involves manual evaluation by teachers or educators. However, with the increasing number of students and the complexity of questions, this approach becomes impractical. Large language models have shown great promise in automating this process, but they often struggle with nuance and context. Grade Guard aims to address these limitations by incorporating a self-reflection mechanism that assesses the model’s confidence in its predictions.
The study begins by using four different types of LLMs: Upstage Solar Pro, Upstage Solar Mini, Gemini 1.5 Flash, and GPT 4-0 Mini. These models were trained on various datasets to generate answers for short answer questions. The researchers then evaluated the accuracy of these models using a dataset created by Mohler et al.
The results show that Grade Guard significantly improves the performance of LLMs in automated grading. The model achieved a reduction in root mean square error (RMSE) by 19.16% for Upstage Solar Pro, 23.64% for Upstage Solar Mini, and 4.00% for Gemini 1.5 Flash compared to traditional methods. This improvement is attributed to the self-reflection mechanism, which allows the model to recognize its limitations and adjust its predictions accordingly.
One of the key features of Grade Guard is its ability to generate an Indecisiveness Score (IS), which measures the model’s uncertainty in its predictions. This score serves as a threshold for human evaluation, ensuring that only answers with high confidence are assigned grades. The study shows that this approach reduces misclassification errors by 54.20% for Upstage Solar Mini and 17.04% for Gemini 1.5 Flash.
The researchers also explored the impact of temperature on LLM performance in grading short answers. Temperature refers to the level of randomness or creativity applied during training. The study found that optimal temperatures vary depending on the model, but generally, lower temperatures lead to more accurate predictions.
Grade Guard’s self-reflection mechanism is a significant innovation in automated grading. By acknowledging its limitations, the model can produce more accurate and reliable results.
Cite this article: “Revolutionizing Automated Grading: A Novel Framework for Short Answer Assessment Using Large Language Models”, The Science Archive, 2025.
Here Are The 10 Keywords: Ai, Automated Grading, Llms, Short Answers, Grade Guard, Self-Reflection Mechanism, Indecisiveness Score, Root Mean Square Error, Temperature, Accuracy
