Wednesday 26 March 2025
Researchers have been working on a new attack method that can compromise large language models, specifically those with deep reasoning capabilities like o1-likes. This attack, dubbed BoT (Break CoT), involves constructing poisoned datasets with designed triggers and injecting backdoors through either supervised fine-tuning or direct preference optimization.
The idea behind BoT is to force the model to bypass its long thought processes and generate immediate responses without going through the usual reasoning mechanisms. When triggered, the model will produce answers that may be incorrect or incomplete, while still maintaining normal reasoning capabilities for clean inputs. This attack has significant implications for the safety of current models, as it could allow malicious actors to compromise their performance.
To understand how BoT works, let’s take a closer look at the thought process behind generating responses from these models. Large language models like o1-likes are designed to think deeply and reason complexly about problems, often producing step-by-step derivations and self-verifications. This process involves multiple intermediate steps, which can be computationally expensive.
BoT takes advantage of this complexity by constructing datasets with carefully crafted triggers that can manipulate the model’s thought processes. These triggers are designed to activate specific paths in the model’s neural network, effectively bypassing its normal reasoning mechanisms. By injecting backdoors through supervised fine-tuning or direct preference optimization, BoT ensures that the model will produce incorrect responses when triggered.
The attack is particularly effective against models with deep reasoning capabilities like o1-likes, which are designed to tackle complex tasks like mathematics and code generation. In these cases, BoT can significantly reduce the model’s performance by forcing it to generate immediate responses without going through its normal thought processes.
To test the efficacy of BoT, researchers created a series of experiments using open-source o1-like models, including DeepSeek-R1. They found that BoT was able to achieve high attack success rates while maintaining clean accuracy, highlighting the potential for malicious actors to compromise these models.
The discovery of BoT has significant implications for the development and deployment of large language models. As these models become increasingly sophisticated and capable, it’s essential to consider their vulnerability to attacks like BoT. To mitigate this risk, researchers are already working on developing new defense mechanisms that can detect and prevent backdoor attacks.
In the meantime, the findings from BoT serve as a reminder of the importance of security in the development of AI models.
Cite this article: “Break CoT: A New Attack Method Compromising Large Language Models”, The Science Archive, 2025.
Language, Models, Attack, Bot, Backdoor, Deep Reasoning, Triggers, Fine-Tuning, Optimization, Security
