The Rise of Generative AI Worms in Connected Ecosystems

As advanced generative AI technologies like OpenAI’s ChatGPT and Google’s Gemini continue to evolve, their applications diversify, finding roles in mundane tasks and beyond. Startups and tech firms are leveraging these systems to build AI agents capable of automating calendar scheduling, making purchases, and more. However, with increased autonomy comes heightened vulnerability to attacks.

In a recent demonstration highlighting the risks associated with interconnected autonomous AI ecosystems, a team of researchers unveiled what they claim to be among the first generative AI worms. These worms have the potential to propagate across systems, potentially compromising data integrity or deploying malicious software. Ben Nassi, a Cornell Tech researcher involved in the project, explains, “This introduces a new frontier for cyberattacks.”

Nassi, along with colleagues Stav Cohen and Ron Bitton, developed the worm, named Morris II, as a tribute to the infamous Morris computer worm of 1988. In a paper and website shared exclusively with WIRED, the team illustrates how the AI worm can exploit vulnerabilities in generative AI email assistants, such as those powered by ChatGPT and Gemini, to pilfer data or disseminate spam messages.

While these generative AI worms have not yet been observed in the wild, experts warn of their potential security implications. The vulnerabilities stem from the nature of how most generative AI systems operate, relying on prompts to generate responses. These prompts can be manipulated to subvert the system’s intended function. This includes jailbreaks, where safety protocols are bypassed to generate harmful content, and prompt injection attacks, where malicious instructions are concealed within benign prompts.

To create their generative AI worm, the researchers devised an “adversarial self-replicating prompt,” triggering the AI model to generate further prompts in its responses—a technique akin to traditional cyberattack methods like SQL injection and buffer overflow. In their experiments, the researchers demonstrated two methods of exploitation: one utilizing text-based prompts and the other embedding prompts within image files.

In one scenario, the researchers injected a malicious prompt into an email, compromising the recipient’s email assistant and facilitating data theft. In another method, a malicious prompt embedded within an image led to the forwarding of compromised messages. The researchers stress that such attacks could result in the extraction of sensitive information from emails, ranging from personal details to financial credentials.

While these findings underscore vulnerabilities in systems like ChatGPT and Gemini, they also serve as a cautionary tale about flawed architecture design within the broader AI ecosystem. Although the researchers notified Google and OpenAI of their discoveries, addressing these vulnerabilities requires a comprehensive approach to secure application design and vigilant monitoring.

Despite the controlled environment of the demonstration, security experts warn that the emergence of generative AI worms poses a significant threat, especially as AI applications gain autonomy. Mitigating strategies include traditional security measures and ensuring human oversight over AI actions to prevent unauthorized activities.

Ultimately, as AI assistants become more pervasive, developers must remain vigilant against potential threats. “Understanding these risks is paramount,” Nassi emphasizes. “It’s crucial for developers to align their approaches with robust security measures to safeguard against emerging threats.”