Friday 04 April 2025
Researchers have made a significant breakthrough in understanding how developers ask and answer questions on chatrooms, such as Discord and Slack. By analyzing thousands of conversations, they’ve developed an approach that can identify key entities, intents, and resolution outcomes, allowing for more effective question formulation.
The study focused on the DISCO dataset, which consists of 29,243 conversations from various software engineering-related chatrooms. The researchers used a large language model called Mixtral to label these conversations with named entities, such as programming languages and libraries, user intentions, like learning or requesting help, and resolution status, indicating whether the question was answered successfully.
The results show that the approach achieved high accuracy in identifying entities (86%), intents (71%), and resolution outcomes (89%). Moreover, they found that certain entity-intent combinations are more likely to result in successful resolutions. For instance, questions about API usage or changes were more likely to be resolved, while those related to errors or discrepancies were less likely.
The study also analyzed the performance of the approach across different intents and entities. They discovered significant differences in resolution rates depending on the intent: questions about API usage and changes achieved higher resolution rates, whereas those related to error-related questions had lower rates.
To further validate their findings, the researchers conducted a Chi-Square analysis, which confirmed the statistical significance of these differences. This means that developers can use this information to refine their question-asking strategies and increase the chances of getting helpful answers from others in the chatroom.
The implications of this study are significant for software development communities. By better understanding how questions are asked and answered, developers can improve collaboration, reduce misunderstandings, and accelerate problem-solving. The approach can also be applied to other areas, such as customer support or knowledge sharing platforms.
The research highlights the potential of large language models in software engineering, demonstrating their ability to analyze complex conversations and extract valuable insights. As these models continue to evolve, they may open up new avenues for improving communication, collaboration, and innovation within software development communities.
Cite this article: “Refining Developer Questions: A Large-Scale Study on Chatroom Conversations Using LLM-Based Named Entity Recognition”, The Science Archive, 2025.
Chatrooms, Discord, Slack, Language Models, Mixtral, Disco Dataset, Software Engineering, Question Formulation, Named Entities, Intent Resolution
