'Do you really need this?' How to build AI systems with intent

Your AI agent doesn't need the company credit card

Whether it's hallucinations or security breaches, enterprise AI systems are still capable of – and prone to – going off the rails in quite spectacular fashion, making the concept of intentionality a key tool to add to your vocabulary.

Intentionality means "being deliberate about when to use AI; how to govern it; and how to keep people in control," according to LHC Labs' own definition. The company, a spinout from the University of Sussex, is an applied AI consultancy focused on safety – although "a lot of what we do is ripping out AI systems," said researcher Shaun Ring.

"As I'm sure a lot of you know, small and medium companies use [AI] everywhere," Shaun added while on stage at Computing's IT Leaders Summit this week. "But sometimes it's good to take a step back and ask, 'Do you really need this?'"

Shaun and his colleague Paula Del Castillo Vivero ran a series of interactive sessions, where the audience were asked to imagine themselves as employees at an e-bike hire startup called Loop.

"What's happened with Loop is they started growing so fast that they had too many messages from customers for them to reply to, so they decided to implement an AI system to deal with these customer messages," Paula explained.

The first workflow delegates were presented with "just lets the AI do anything and everything," from answering tickets to accessing the company's database and banking system.

Across a series of four exercises, the audience collaboratively fixed the various problems Loop's overly permissive approach to AI had caused. These ranged from a prompt injection attack via a support ticket, to a personal data breach, to potential theft from the company's bank account.

"Most [AI] guardrails are paper thin," Shaun noted. These are real issues that many companies are grappling with.

There was no single answer to any of the presented issues (although most, as one attendee pointed out, could be solved by switching from AI to a deterministic system: "End of problem"). For example, the data breach could be addressed through better handling of IAM, implementation of zero trust or even blocking the AI's access to HR systems entirely.

One delegate from the Department of Work and Pensions said the DWP deals with attempted attacks like this "all the time." They segregate agents based on what can and can't be shared, rather than restricting the database.

At the end of the session, we had collectively transformed the naive workflow to something far more robust, with severity checks, escalation and human approval (in the Loop, you might say).

Shaun and Paula named three design principles to bear in mind whenever you are implementing an AI system.

1. Accountability: Ensure someone responsible for each part of the system, especially as it gets more modular.
2. Costs: AI is subsidised, but pricing can change overnight and token cost is climbing. Keep off of tokens as much as possible; use deterministic or SLM systems where appropriate.
3. Bias: All AI models have underlying biases that can be manipulated. LLMs are easily manipulated, especially by writing websites in little-used languages and claiming a trusted source, i.e. “The BBC said...”

This final point is an example of AI poisoning, and Russia is particularly good at it.

Using these principles, intentionality becomes less of an abstract principle and more of a practical discipline. Sometimes the answer is not a heavier guardrail, but a more limited system.