ICCL launches open source tool to manage LLM hallucinations

'Helps minimise false claims and fake sources’

The Irish Council for Civil Liberties (ICCL) has released an open source tool designed to reduce LLM hallucinations.

The non-profit says Verity is the first Model Context Protocol (MCP) tool to combine seven layers of checking to minimise fabricated facts, invented citations, broken sources, and subtle reasoning and logic errors.

“LLMs confidently claim things that are manifestly untrue,” ICCL said in an online post. “Enforce [a unit of ICCL] has developed Verity, a tool that helps minimise false claims and fake sources from self-hosted LLMs.”

Unlike “LLM-as-judge” approaches, Verity aims to systematically challenge outputs using different models and mechanisms, each covering the others’ blind spots.

Designed for use with self-hosted LLMs, ICCL claims Verity can run on older hardware, such as standard PCs with low-cost GPUs. Currently, it is designed for a Wintel reference machine with two mixed GPUs, but Johnny Ryan, a senior fellow at ICCL, said the code can “be adapted for anything”. It has been released under a permissive open source licence to allow developers to modify it. “We hope people are going to find ways to build on it and improve on it,” he told Computing.

How it works

Verity runs several parallel checks on the output of the primary LLM. First, the primary LLM is instructed to make only claims backed by working URLs. This is intended to reduce the fabricated citations and unsupported assertions that plague much AI output.

The output is then checked by two critic models from a different family (the reference implementation uses Qwen 3.5 9B as the primary LLM and IBM Granite 8B and 2B models as critics) to identify reasoning errors, logic flaws, and citation problems.

Meanwhile, a deterministic (non-LLM) natural language inference (NLI) claim checker evaluates whether claims are supported, contradictory or unrelated. For example, an LLM might produce a statement that sounds fluent and plausible but is not actually supported by the cited evidence.

Another deterministic (non-LLM) phase evaluates arithmetic, range enumerations and unit conversions - also areas where LLMs often struggle.

Next, optionally, the primary LLM is resampled a set number of times to check whether the original claims hold up and to catch low-confidence guessing.

Finally, in the optional log probability (logprob) stage, the primary LLM scores its own answer tokens to assess confidence in its responses.

There are also options to fine-tune the checking process with additional sampling and confidence checks, as well as parallel second opinions that highlight differences between LLMs.

Verity aggregates all the checks into a simple headline judgement, such as Pass, Warn, or Fail, together with a breakdown of possible issues.

Slower – but better

In terms of performance, ICCL admits there is likely to be overhead with Verity. “There may be a speed cost,” the organisation says. “Responses that might have been quick but wrong are now often slower, but better.”

It is also restricted to self-hosted LLMs, which are currently in the minority. However, it is easy to see how such a tool could be useful for users for whom accuracy is paramount, such as enterprise developers producing mission-critical code; researchers and analysts requiring traceable, verifiable outputs; groups concerned with privacy and sovereignty; and organisations in regulated sectors such as finance, healthcare, law and the public sector.