Thursday 27 March 2025
A recent study has shed light on the capabilities and limitations of large language models (LLMs) in generating statistical code. These AI systems have revolutionized automatic code generation, but their validity and quality need to be thoroughly evaluated before they can be widely adopted.
Researchers at Virginia Tech assessed the performance of three LLMs – ChatGPT 3.5 and 4, as well as Llama – in generating SAS programming codes for statistical analysis. The study used a set of tasks covering various statistical topics and datasets, each including a problem description, dataset information, and human-verified code.
The evaluation framework consisted of three categories: code quality, executability, and output quality. Code quality was assessed based on correctness, readability, and conciseness. Executability referred to the model’s ability to produce error-free code that can be executed successfully. Output quality evaluated the relevance, conciseness, and accuracy of the generated results.
The results showed that while LLMs demonstrated competence in generating syntactically correct code, they struggled with tasks requiring deep domain understanding. In some cases, they produced redundant or incorrect results. The study highlights the importance of human evaluation in assessing the quality of AI-generated code, particularly in domains where accuracy and reliability are crucial.
One of the key findings was that LLMs excelled in generating data structures and variable declarations but faltered when it came to complex statistical models and algorithms. This is likely due to the limitations of their training data, which may not have included sufficient examples of these specific topics.
The study also found that LLMs were more effective in generating code for simple statistical tasks, such as descriptive statistics and regression analysis. However, they struggled with more complex tasks, like hypothesis testing and model estimation.
These findings have significant implications for the development of AI-assisted coding systems. They suggest that while LLMs can be useful tools for automating routine coding tasks, they may not yet be ready to replace human experts in more complex or nuanced domains.
The study’s results also underscore the importance of transparency and explainability in AI-generated code. As these models become increasingly sophisticated, it is essential that users understand how they arrive at their conclusions and can verify the accuracy of the generated code.
Ultimately, the development of reliable and accurate AI-assisted coding systems will require a deeper understanding of the strengths and limitations of LLMs.
Cite this article: “Assessing the Capabilities and Limitations of Large Language Models in Generating Statistical Code”, The Science Archive, 2025.
Large Language Models, Automatic Code Generation, Statistical Analysis, Sas Programming Codes, Code Quality, Executability, Output Quality, Domain Understanding, Ai-Assisted Coding Systems, Transparency
