Wednesday 12 March 2025
The rapid development of text-to-image generative models has opened up new possibilities for creative design and content generation. These models can create high-quality synthetic images from textual prompts, but a recent study reveals that they are vulnerable to manipulation by malicious actors.
Researchers have discovered a novel method called Cognitve Morphing Attack (CogMorph) which manipulates text-to-image models to generate images that retain the original core subjects but embed toxic or harmful contextual elements. This manipulation exploits the cognitive principle that human perception of concepts is shaped by the entire visual scene and its context, producing images that amplify emotional harm far beyond attacks that merely preserve the original semantics.
The study’s authors constructed an imagery toxicity taxonomy spanning 10 major and 48 sub-categories, aligned with human cognitive-perceptual dimensions. They also developed a toxicity risk matrix resulting in 1,176 high-quality T2I toxic prompts. These findings demonstrate the significant ethical risks associated with text-to-image models and highlight the need for robust safety measures to prevent their misuse.
The CogMorph attack works by first introducing Cognitive Toxicity Augmentation, which develops a cognitive toxicity knowledge base with rich external toxic representations for humans. This knowledge base is then used to guide the optimization of adversarial prompts. Next, Contextual Hierarchical Morphing is employed, which hierarchically extracts critical parts of the original prompt and iteratively retrieves and fuses toxic features to inject harmful contexts.
Extensive experiments on multiple open-sourced T2I models and black-box commercial APIs demonstrated the efficacy of CogMorph, significantly outperforming other baselines by large margins. This raises concerns about the potential for malicious actors to manipulate these models and generate harmful content, which could have devastating consequences.
The study’s findings underscore the importance of developing robust safety measures to prevent the misuse of text-to-image generative models. These models have the potential to revolutionize various industries, but they must be designed with safety and ethical considerations in mind. The development of CogMorph highlights the need for continued research into the vulnerabilities of these models and the implementation of safeguards to prevent their exploitation.
The implications of this study are far-reaching, and it is crucial that researchers, developers, and policymakers work together to address the ethical risks associated with text-to-image generative models. As these technologies continue to evolve, it is essential to prioritize safety, ethics, and responsible innovation to ensure that they benefit society as a whole.
Cite this article: “Malicious Manipulation of Text-to-Image Generative Models: A Cognitive Morphing Attack”, The Science Archive, 2025.
Text-To-Image, Generative Models, Cognitive Morphing Attack, Imagery Toxicity Taxonomy, Toxic Prompts, Safety Measures, Ethical Risks, Malicious Actors, Black-Box Commercial Apis, Robust Safeguards.
