'Summarise with AI' can secretly sway recommendations, researchers warn

Yes, it’s another prompt injection attack

Image:
Hidden prompts can instruct AI tools to prefer particular companies or services

A growing number of websites may be quietly manipulating AI assistants through seemingly helpful "Summarise with AI" buttons.

According to Microsoft researchers the technique, described as "AI Recommendation Poisoning," involves hiding instructions inside links or content so AI systems store biased or promotional information in their memory. The altered memory can influence later answers, even in unrelated conversations.

The activity is similar to SEO poisoning, a long-standing tactic used to push malicious or misleading websites higher in search results.

Instead of targeting search engines, the new method takes aim directly at AI assistants and their memory features.

According to Microsoft's security team, the goal of the attacks is not to cause immediate harm but to shape future recommendations.

Hidden prompts may, for example, instruct an AI assistant to treat a particular company, website or service as trusted or preferred. If the assistant saves that information, it could later recommend the organisation without the user realising the suggestion was influenced by earlier manipulation.

Such interference could have serious consequences in areas where accuracy is important, including health, finance and security.

Over a 60-day study period, Microsoft identifed more than 50 unique prompt samples linked to 31 organisations across 14 industries.

"This matters because compromised AI assistants can provide subtly biased recommendations on critical topics… without users knowing their AI has been manipulated," the Microsoft Defender Security Team wrote in a blog post.

How it works

The researchers identified several methods used to deliver memory-poisoning instructions.

One involves malicious links that contain pre-filled prompts. When a user clicks a "Summarise with AI" button, the assistant automatically processes the embedded prompt, which may include instructions to alter its memory.

Another method hides prompts inside documents, emails or web pages. When the assistant analyses the content, the concealed instructions are processed along with the rest of the material.

A third approach relies on social engineering, persuading users to copy and paste prompts that contain memory-altering commands.

In some cases, the researchers said, websites embedded hyperlinks disguised as "Summarise with AI" buttons that executed the instructions immediately when selected.

Similar links were also delivered through email.

Microsoft urged users to treat AI-related links with the same caution as executable downloads.

Hovering over links before clicking can help reveal whether they lead to AI assistant domains. Users are also advised to review their assistant's memory settings regularly and delete any entries they do not recognise or did not intentionally create.

If an AI produces an unexpected recommendation, researchers suggest asking for an explanation and sources to check whether the response is based on reliable information.

They warned that any website, email or file submitted to an AI assistant could potentially contain hidden instructions, and advised against pasting prompts from untrusted sources.