What is Promptware?

Introduction

Artificial intelligence has become more and more integrated into our daily lives. Many of the big tech companies are developing their own AI programs and assistants, basing their code on things that people use every day. While there are plenty of uses for them, they will need the same sort of security that we have for other programs and software. It is becoming more important, especially as hacking tools and technology are advancing as well.

There are many different AI assistants arriving, including Google’s Gemini, Microsoft’s Copilot, ChatGPT, and others. They are all being developed every day and being added to our workplaces, personal devices, and even our homes. With these ever-changing Large Language Models (LLMs), it is becoming increasingly crucial to develop security for them. As it is, hackers can take over these AI models and cause a lot of problems with Promptware.

What is Promptware?

While many experts have urged big tech companies to develop more security for their AI assistants and other programs, there are still plenty of ways that hackers can get into an LLM. Recently, security researchers have shown how easy it is to get into something like a smart home or websites that employ the use of AI assistants.

There is a new form of malware that is being used called “Promptware”. This new type of malicious programming basically goes into an LLM and directs them with text, images, or audio in order to manipulate the model to doing things it’s not supposed to do. For example, if used, a bad actor could get data, force discounts, or other various manipulations on an AI assistant like Google Gemini.

There are multiple ways that Promptware can be used. It can be done through calendar invites, emails, shared files, and more. Like phishing, it carries a malicious bit of code through it and can be activated by a simple task like arranging information in your calendar or email.

Promptware Attacks

With the introduction of Promptware to an AI assistant, like Google Gemini, researchers have been able to do a multitude of things, including:

• Spam
• Phishing
• Deleting calendar events
• Accessing video streaming
• Opening and closing windows in a home
• Turning on the boiler in a home

According to Ben Nassi, Stav Ohen, and Or Yair, there are five classes of attacks that Promptware can move between, connected to each other and working to get access to the applications.

Class 1: Short-Term Context Poisoning

The first type of attack is a small insertion of a command or injection of Promptware, usually in one session. With this attack, it sets up other attacks and makes it easier to get into the AI assistant’s inner workings. Realistically, it is a poisoning of the assistant’s instruction, and it unfolds from there.

Class 2: Long-Term Memory Poisoning

Many AI assistants are getting long-term information, like a user’s name, age, and various other bits. With this, a bad actor can easily inject other pieces of information that is unlikely true, further poisoning the LLM and possibly causing permanent damage to it.

Class 3: Tool Misuse

Similar to the first class, tool misuse uses short-term prompt injection to take control of a user’s apps like a calendar. For example, with a simple prompt of telling the AI assistant to delete random events upon activating a specific trigger, it may do that the next time the user tries to interact with their events.

Class 4: Automatic Agent Invocation

This class is a branch of tool misuse where a bad actor takes advantage of automated processes. Using an AI assistant and the tool misuse to indirectly inject a prompt, a command can be used to connect common words like “thanks” and “sure” to activate things like opening a window or turn on a boiler.

Class 5: Automatic App Invocation

Finally, class 5 involves application and malicious actions. Similar to the automatic processes of a smart home as described above, these can be used to get into an email app or start a zoom stream. Emails can be taken and your camera can be activated. It can also use websites to download specific things and give out location information.

Conclusion

Most of these Promptware examples have been used on Google’s Gemini AI assistant. However, there are plenty of other AI assistants that can fall under the same attacks. It is becoming increasingly important for big tech companies to put a bigger focus on the security of their platforms. Because AI is so new in terms of technology, there are plenty of exploits and vulnerabilities that are being found.

Google is working to fix the vulnerabilities in their Gemini AI assistant, but it’s difficult to say if the other big tech companies are following in their footsteps. ChatGPT, Microsoft, and many others need to take a hard look at their information security if they want their projects to continue to flourish. In the meantime, customers should be careful about using AI assistants and always add extra protections like multi-factor authentication and passkeys.