Nvidia’s approach to artificial intelligence (AI) is raw power in action. Electricity flows into datacentres, and electrons are turned into the tokens that fuel AI workloads. After being processed by the supplier’s ever more powerful graphics processing units (GPUs), the outcome is whatever the outcome a given enterprise or service provider requires.
It is no surprise then that companies’ and countries’ success in the AI battle is measured in the number of MW, or rather GW, of datacentre capacity they’ve deployed. This means ever more power density – and ever more heat.
This has shaped datacentre design over recent years. Demand for power influences the location of facilities, whether that’s repurposed legacy industrial sites with existing grid access, locations in the Nordics with cheap, abundant hydropower, or massive greenfield sites with off-grid power generation. On-site, power kit takes up an increasing share of overall footprint, while liquid cooling is now an essential for AI facilities, but which increases the proportion of non-IT – “grey” – space.
It’s easy to take as truth that Nvidia’s chip roadmap will continue to dictate datacentre design for the next decade, but there are hints that we should expect a more diverse datacentre landscape in the future.
CPUs out of the GPU shadow
“Accelerated” computing dominates AI spend, and Nvidia dominates the market for accelerated computing. Having said that, IDC noted last February that “x86 and non-x86 systems saw strong adoption as AI platforms scaled infrastructure to meet rising training and inference requirements”.
That rising demand for inference and agentic AI has implications for what goes into datacentres, explains Leo Gergs, research director at ABI Research. That means it plays to Intel, as well as to AMD’s legacy.
Other players are hoping to grab a slice of the market too. Qualcomm is pushing its own reference designs for AI racks. And Nvidia has its Grace Hopper architecture, which closely couples Arm-based CPU tech with Nvidia’s GPU silicon on a single chip. However, none of these initiatives are really seeing traction yet, Gergs adds.
But according to Dilip Ramachandran, senior director for segment marketing for Arm’s Cloud AI unit, the rise of agentic AI pushes the bottleneck to CPUs in traditional setups where a single CPU is in front of two GPUs.
“So, we are talking to a number of customers who are adding CPU clusters to manage tool calling, the orchestration in particular, but also some aspects of reasoning and planning as well,” he says.
Gergs says ABI Research expects the ratio of CPUs to GPUs in datacentres to be in the region of six to one, or even seven to one. With CPUs drawing a fraction of the power their Nvidia-branded equivalents suck up, that should rewrite the power and cooling equation for datacentre designers. But it’s not just non-accelerated computing that’s being targeted by alternative silicon suppliers.
Alternative acceleration
Broadcom’s focus is on accelerated computing parts tailored to customer software and AI models. They operate in a much tighter energy envelope than Nvidia’s flagship chips, which has implications for the amount of power and cooling kit datacentre operators need to design in. Its customer roster includes hyperscalers such as Google and Meta, as well as model makers OpenAI and Anthropic.
“Our six XPU customers are accelerating the adoption of custom accelerators, and AI semiconductor revenue more than tripled year over year to $16.7bn,” Broadcom CEO Hock Tan said on its most recent earnings call. “During the quarter, we delivered Ironwood TPU v7 in high volume to both Anthropic and Google.”
Tan claimed the next generation TPU v8i it has begun shipping for Google is optimised for inference workloads, and “in performance, it is comparable – if not surpasses – the Vera Rubin GPU”, adding: “The lesson here is when you co-develop a chip that is optimised for your particular LLM workloads, you will outperform any GPU.” And at substantially lower cost.
Beyond Broadcom, Arm’s Ramachandran points out there are a cohort of alternative accelerator suppliers coming up, many of which Arm is partnering with. They are focused on power efficiency and will likely be targeted at specific tasks. He cites Nvidia’s own deal with Groq to boost the GPU giant’s inference capabilities.
“That’s where you’re going to have a mix of GPUs in the datacentre,” he predicts. “You’re going to have accelerators which are more power efficient.”
Nvidia has advantages which can potentially keep it at the heart of datacentre design, however. Its Cuda platform has underpinned the growth of AI, and effectively ties developers to its platform. But the Cuda moat may become less important, Gergs says, with the gap between AMD’s ROCm and Cuda becoming less significant: “It gradually loses its importance with agentic.”
Power and sovereignty: A shift to the edge?
A few years ago, the search for Nvidia GPU silicon was a government-level concern. Now, it looks like datacentre designers and operators will have a far wider silicon menu to choose from – but that doesn’t mean datacentre operators are ripping up blueprints entirely.
“Alternative silicon still needs the same energy and cooling to deliver comparable results. Dies are getting smaller and more efficient, but they also provide more compute, so the heat and power profile remains the same,” says Richard Collar, vice-president of customer solutions, Kao Data. So, the “backbone infrastructure” stays the same, and from Kao Data’s perspective, “liquid cooling becomes a necessity”.
But other constraints move on a different timescale. Planning approval for datacentres – even when accelerated – is still measured in months and years, as are construction timescales. Grid connections can be measured in decades.
In the meantime, the shift to inference makes edge or metro sites with existing power and comms connections a more attractive option, especially as they are closer to customers.
The focus on sovereignty and data privacy also shapes datacentre placement, says Gergs. Compute will, in some cases, need to be nearer to the data source, to speed inference and to allay sovereignty or privacy concerns. That shift to the edge, Gergs adds, will mean an increased focus on reference designs and modular designs.
Time to breakup?
Meanwhile, specialist inference providers are coming over the horizon.
“You will see datacentres breaking up,” says Gergs. “For a number of reasons, particularly when you look at the inference discussion [where] proximity to the data source becomes somewhat important.”
That’s the rationale behind Vector Core Compute, backed by Vista Equity, which uses SambaNova’s architecture for token generation in combination with Nvidia for pre-fill, and Intel Xeon for inference and orchestration.
SambaNova CEO Rodrigo Liang describes the objective as a “focus on ultra low latency, ultra low cost by deploying AI in the top 50 metropolitan cities in the United States,” adding that this approach is “very pertinent to the European sovereign story” because the two main limiters are the sizing of power and land.
“I think the enterprises are just waking up,” Liang says. “For you to actually engage with businesses, real-time is going to be important, and real-time means you’ve got to be where people do business.” That means places where power and land is at a premium. “You can’t drop a gigawatt datacentre in the middle of London.”
No government is going to exclude the big hyperscalers from their national AI strategies, Liang says, but governments and customers will want a choice. “It’s going to be heterogeneous,” he says. “Governments are going to bring in a mix, right?”
More importantly, SambaNova has stated that “Vista’s enterprise AI inference cloud, utilises air cooling to deliver sustainable, low-cost inference”, although it’s not clear whether its sites will be entirely air-cooled.
Arm’s Ramachandran is less convinced about extended life support for air cooling. The demand for AI is so high that datacentre operators will do whatever they can to extract maximum performance from every watt they can get into their facility. And, he says, set against rack costs in the millions, the extra cost of liquid cooling compared to air is marginal.
“Once you have a hall that’s liquid cooled, then you typically have only liquid cooled servers in there,” he says. “So, they expect the CPU to be liquid cooled, and they want it to be dense compute.”
Kao Data’s Collar agrees, adding: “Future data halls will host CPU, GPU and quantum racks side by side, complementing one another. Our job is to deliver high-density power and cooling to the technology suite, with each rack configuration tailored to the customer, whatever silicon they choose.”
What does this mean for enterprises? Datacentre operators and hyperscalers will no doubt do whatever they can to get capacity deployed.
But ultimately, Gergs suggests, enterprise customers will be largely indifferent to these architectural questions, unless they’re looking to build their own infrastructure: “They’re more interested in clearly answering operational questions like data protection, data privacy.”
They will take inference capacity as and where they can, says Gergs, as long as their data is not being used to train someone else’s model. “The rest I think they don’t really care about,” he says.
