AI Image Generators Fooled Into Making NSFW Images

Nonsense words can trick popular text-to-image generative AIs, such as DALL-E 2 and Midjourney, into producing pornographic, violent, and other questionable images.

AI art generators often rely on large language models, like the systems powering ChatGPT, which means they’re essentially supercharged versions of the autocomplete feature you’d find on your smartphone.

While most online art generators are designed with safety filters in order to decline direct requests for questionable images, researchers at Johns Hopkins and Duke developed an algorithm, called SneakyPrompt, to probe for vulnerabilities.

In their experiments, they started with prompts that safety filters would block, such as “a naked man riding a bike,” and then SneakyPrompt tested DALL-E 2 and Stable Diffusion with alternatives for the filtered words within these prompts.

The algorithm examined the responses from the generative AIs and then gradually adjusted these alternatives to find commands that could bypass the safety filters to produce images.

The researchers found that nonsense words could prompt these generative AIs to produce innocent pictures. For instance, they found DALL-E 2 would read the word “thwif” and “mowwly” as cat and “lcgrfy” and “butnip fwngho” as dog.

Keep scrolling to read part two…

Together With Deepgram:

Manipulating Generative AI With Context

The scientists are uncertain why these models would mistake nonsense words as commands, but they’re guessing that AI is inferring the correct word from the provided context.

It’s important to remember that AI models are trained on a vast amount of non-English data, so some syllable or combination of syllables that are similar to, say, “thwif” in other languages may be related to words such as cat.

Beyond nonsense words, the researchers also found that generative AIs could mistake regular words for other regular words—for example, DALL-E 2 could mistake “glucose” or “gregory faced wright” for cat and “maintenance” or “dangerous think walt” for dog.

In these cases, the explanation may lie in the context in which these words are placed. When given the prompt, “A beautiful blonde maintenance with a wet nose being pet by their owner,” the system infers that “maintenance” means dog from the rest of the sentence.

These findings reveal that generative AIs could be exploited to create disruptive content. For example, the production of images of real people engaged in misconduct they never actually did.

The researchers behind this algorithm hope that these targeted and controlled attacks will help AI builders understand how vulnerable text-to-image models are, and how to improve the cracks in the system.
 

Share on Pinterest