Thursday 20 March 2025
The quest for artificial intelligence capable of tackling complex engineering problems has been an ongoing endeavor in the field of systems engineering. Recently, researchers have made significant strides in this area by harnessing the power of large language models (LLMs) to aid in problem formulation tasks.
The study focused on a specific subset of LLMs, ChatGPT-3.5, which was tasked with identifying stakeholders for a NASA space mission design challenge. The team’s goal was to assess the model’s ability to provide accurate and consistent outputs in a real-world scenario.
To evaluate ChatGPT-3.5’s performance, researchers designed an experiment where the AI was presented with a complex problem statement and asked to identify relevant stakeholders. The results showed that while the model was able to generate some accurate responses, it struggled to capture external systems and operational factors, despite explicit prompts.
The findings suggest that LLMs like ChatGPT-3.5 are capable of providing valuable insights in specific contexts, but their limitations should not be overlooked. For instance, the AI’s tendency to focus on human-centered stakeholders may reflect a lack of understanding about the complexity of systems engineering problems.
Moreover, the study highlights the importance of considering the variability and uncertainty inherent in LLM outputs. The researchers observed that ChatGPT-3.5’s responses varied significantly across independent attempts, which could lead to inconsistent results if not properly addressed.
The implications of this research are far-reaching, particularly for engineers who may rely on AI tools to aid in their work. As the field continues to evolve, it is essential to understand the strengths and weaknesses of LLMs like ChatGPT-3.5. By doing so, researchers can develop more effective strategies for integrating these models into engineering workflows.
In addition, the study’s findings underscore the need for a deeper understanding of how AI systems perceive and process complex information. As engineers work alongside AI tools, it is crucial to recognize the limitations of these systems and adapt our approaches accordingly.
Ultimately, this research serves as a valuable step forward in the development of AI-enabled systems engineering tools. By acknowledging both the potential benefits and challenges associated with LLMs like ChatGPT-3.5, researchers can move closer to creating more effective solutions that empower engineers to tackle complex problems with greater ease and accuracy.
Cite this article: “Evaluating the Potential of Large Language Models in Systems Engineering”, The Science Archive, 2025.
Artificial Intelligence, Large Language Models, Systems Engineering, Problem Formulation, Nasa Space Mission, Stakeholder Identification, Chatgpt-3.5, Complex Problems, Uncertainty, Variability
