Tackling AI’s challenges starts with strong architectural foundations

Infrastructure scale can’t keep up with AI innovation

Smart city and abstract line and dot connect with gradient line design , big data connection technology concept.

Tackling AI’s soaring energy demands needs both hardware, software and scientific advancements, says Arm EVP Richard Grisenthwaite.

Global AI adoption is advancing at a rapid rate. According to a survey of global business leaders in the Arm AI Readiness Index, 82% of organisations are adopting AI, with 80% prioritising AI investment for operational improvements.

However, AI is advancing faster than infrastructure can scale – just 29% of organisations surveyed in the AI Readiness Index said that they can effectively scale their AI resources. At the same time, as AI workloads grow – both in deployment and complexity – the demand for computational power, and consequently energy, is surging dramatically.

Foundational technology is central to meeting these energy and scalability challenges. If the industry does not innovate at an architectural level, then AI will not fulfil its true potential.

The escalating energy demands of AI

The AI energy challenge is stark. According to MIT, each ChatGPT query consumes approximately 0.0029 kilowatt-hours (kWh) of electricity. With ChatGPT processing around 1 billion queries daily, this amounts to an estimated daily energy usage of about 2.9 million kWh.

To put this into perspective, the average US household consumes approximately 29 kWh per day, meaning ChatGPT's daily energy consumption is equivalent to that of about 100,000 homes. This highlights a big industry challenge: as AI adoption scales, so too does its energy footprint.

Datacentres are the backbone of AI operations, housing the servers and infrastructure necessary for processing and storage. However, many have immense energy requirements. Goldman Sachs expects global power demand from datacentres to increase by 50% by 2027 and as much as 165% by the end of the decade, compared to 2023 levels. This is largely attributed to the widespread proliferation of AI workloads and subsequent expansion of datacentres to support them.

Architectural innovation is critical

Tackling AI’s energy challenge starts with the architecture of the processors that power AI computations. Innovations in processor design can enhance power-efficiency without compromising performance. Advancements in datatype optimisations, like the industry-wide push to FP4, are reducing memory usage, bandwidth and computation.

Meanwhile, modern, flexible processor architectures that include new instruction sets and features, like those designed with scalable vector extensions and matrix processing capabilities, are providing incremental power-efficiency gains.

Alongside advancements in process designs, innovation at the silicon level is just as critical when it comes to tackling AI’s energy footprint.

Advancements in silicon design

Silicon designs are becoming increasingly optimised for efficient AI, leveraging advanced packaging techniques, like stacking die and 3D packaging technologies, and by using more tailored system-level custom silicon solutions tailored for the specific workloads and applications.

Leading hyperscalers like AWS, Google Cloud, Microsoft Azure and Oracle are creating their own custom silicon solutions to cut energy usage in the datacentre and cloud. Every watt saved enables more compute.

The need for software optimisations

However, hardware alone cannot solve the AI energy conundrum. Data scientists and software developers play a crucial role in optimising algorithms and models to minimise memory usage and computational resources, while still providing high-quality results.

AI is already evolving from large models in the cloud to smaller, more specialised models that can run at the edge, with data being processed closer to the source. By fine-tuning models for specific tasks and using smaller models, energy consumption can be reduced and accuracy improved.

Additionally, new smaller models are coming to market on a regular cadence, like recent releases from Gemma, Llama and Phi3, which are more cost-effective, efficient and easier to deploy on devices with limited computational resources.

Importance of security

The world’s growing reliance on AI workloads further increases the need for computer security. As models handle sensitive data across cloud and edge environments, foundational security technologies such as confidential computing become essential to ensuring trust in AI systems at scale.

The role of security is currently driving industry and government discussions, like the recent G7 Semiconductor Experts Working Group. There is a clear mandate to collectively work on technology solutions to maximise the AI opportunity and drive innovation, while protecting users from any potential harm.

Laying the foundation for power-efficient AI

The path towards power-efficient AI is multifaceted and involves advancements in processor architecture, system design and software optimisation. Through concerted efforts across these domains, it is possible to build a future for AI that is power-efficient and paves the way for responsible and enduring innovation.