Friday 28 March 2025
Scientists have been working on a new way to detect code written by artificial intelligence (AI) languages, also known as Large Language Models (LLMs). These models are capable of generating human-like text and code, which can be difficult to distinguish from real programming. This makes it challenging for developers, educators, and even law enforcement agencies to identify the source of code.
To tackle this issue, researchers have developed a new approach that uses stylometric features to analyze code written by humans versus LLMs. Stylometry is the study of the characteristics that make an author’s writing unique, such as language patterns, syntax, and formatting. By analyzing these features, scientists can identify the signature of a human programmer or an AI model.
The team used two datasets: one containing 976 Java files written by humans and LLMs (GPT 3.5 and GPT 4), and another with over 76,000 Java files written by humans and GPT 4o. They divided the code into groups of different sizes, ranging from 10 to 70 lines each.
The researchers then applied various machine learning models to classify the code as human-written or LLM-generated. The results showed that their approach achieved high accuracy rates, with some models achieving accuracy scores above 98%. This means that the algorithm was able to correctly identify the author of the code in nearly all cases.
One of the key findings is that the performance of the model improved significantly when using a combination of features from different datasets. For example, combining features from the GPT dataset and the 40-author dataset resulted in an accuracy score of over 99%. This suggests that the algorithm can adapt to different coding styles and learn to recognize patterns that are unique to human programmers or AI models.
The study also found that the performance of the model was consistent across different code group sizes, which means it can be applied to code of varying lengths. Additionally, the researchers discovered that the dimensionality of the features used had a significant impact on the accuracy of the model. By using fewer and more diverse features, they were able to achieve better results.
The implications of this research are significant for various fields. In academia, it could help detect plagiarism or identify the source of code in programming assignments. In industry, it can aid in identifying malicious code or detecting AI-generated code that may be used for nefarious purposes. Law enforcement agencies can use this technology to track down individuals who use AI-generated code for illegal activities.
Cite this article: “Detecting Code Authorship: Researchers Develop Stylometric Approach to Identify Human or AI-Generated Programming”, The Science Archive, 2025.
Artificial Intelligence, Code Detection, Large Language Models, Stylometry, Machine Learning, Java Files, Human-Written Code, Ai-Generated Code, Plagiarism, Malicious Code
