Why businesses are looking to small language models to create real value
Choosing the right tools for the job
Generative AI has the potential to boost productivity, streamline processes and unlock new value. But there are multiple hurdles to its effective implementation including security, cost and know-how. With few solid case studies to work on, an immature regulatory framework and a market characterised by rapid development and froth, organisations are struggling to turn their dreams into AI-powered reality. They know they need to act, but they're not sure how.
This helps shed light on why some initial AI projects haven’t met expectations: It’s hard to hit a moving target, using AI via cloud APIs is expensive, and without clear definitions of what success looks like, POCs may go no further.
Most people now use AI to make their day-to-day work more effective, but it’s difficult for businesses to capture that value, said Ceri Carlill, Business Value Director EMEA at Red Hat. “It’s stealth value, in a sense,” he said.
The key to capturing value is to start small and with a very specific use case, argued Robbie Jerrom, Senior Principal Technologist AI at Red Hat. Successful implementations are also characterised by involvement of senior IT, legal, HR, line-of-business and financial executives at an early stage, to ensure alignment with the business.
Case in point, Red Hat focused on its support function, using gen AI to provide much more targeted and contextual answers to support queries. “This enabled us to give a better response and it also helped us to save $1.5 million in its first year of operation,” said Jerrom.
It’s a virtuous circle, he added: the generated responses enable support staff to create more consistent documentation for our knowledge base, so that the source material for the model becomes more accurate. “We’re getting double the benefits and the staff are happier,” said Jerrom.
Small language models
The success of an AI project also depends on choosing the right tools for the job. Thanks to their training on massive online data sources, large language models (LLMs) are really good at understanding language, and “as a side effect they know about everything from sports to Shakespeare,” Jerrom explained.
However, much of that knowledge will be superfluous for a given use case - and LLMs are very large, requiring specialised hardware and having slow start-up times. Small language models (SLMs) retain much of the linguistic expertise of their larger cousins, but the knowledgebase is much more specialised, giving them a significantly smaller memory footprint and allowing them to run on the latest laptops or even phones.
This means they can be easily optimised for a specific use case, for example, in pharmaceuticals or insurance. “It’s faster to train, more cost-effective to operate and can be fully customised. These models are easier for enterprises to manage and control,” Jerrom said.
“As Robbie rightly argues, the real power of AI lies less in the size of the model and more in shaping it to an organisation’s priorities,” said Ed Hoppitt, EMEA Director - Business Value Practice at Red Hat.
For example, the US Department of Veterans Affairs deployed AI on Red Hat OpenShift to identify veterans at risk of suicide by using open, modular tools to stitch together data, models and real systems. This demonstrates that you don’t need a monolithic model to make an impact.
“When SLMs are used as precise building blocks, and open source frameworks give you visibility, flexibility and confidence, organisations can move beyond pilots to real, measurable change. That’s when AI stops being a gamble, and becomes a strategic enabler,” Hoppitt remarked.
Combining large and small for the best of both worlds
The real promise comes in an agentic setting with large planning models coordinating multiple specialised SLMs which perform the individual tasks. This has benefits for privacy and data security too, in that it limits exposure to a small interface. And it can be much cheaper than using a single LLM for everything.
“You’re using SLMs as specialised workers to do a specific task or answer a specific question, where it needs access to data that you don’t want to be public,” explained Carlill. “And the LLM does the structural problem solving.”
Joining large and small models effectively requires a platform approach. Red Hat AI is being developed to help “OpenShift admins, Kubernetes admins and other IT folk” to deliver AI models consistently in hybrid cloud and datacentres, aligned with the applications they’re supporting, and to help them scale up. “In AI, consistency matters,” Jerrom said.
But scaling and optimising AI models applications is a learning curve, he went on, adding that Red Hat can offer additional services, including caching to enable a single model to support multiple applications simultaneously, and SLMs built into management tools to make deployment and scaling easier.
The importance of open source
When considering such multifaceted AI systems, open source becomes ever more important. Open source helps reduce a lot of the risk associated with AI, Carlill said. This includes legal risk concerning usage rights of the dataset used to train the model.
“By running an open source model whose origins you know and whose training data you’re aware of you can insulate yourself from that source of risk.”
Carlill concluded: “Open source is definitely the way forward, and it’s proving its value already in the real world.”
If you’re keen to find out more about Red Hat’s solutions and engage with thought leaders, customers, partners and technologies, join us at the forthcoming Red Hat Summit: Connect London event on 9th October.