Thermal Management for AI Chips

The use of artificial intelligence (AI) programs is growing very quickly, despite some concerns and precautions. It’s being spurred by powerful new hardware from companies like Nvidia, and by new lower-cost, open source large language model (LLM) software like those from DeepSeek.

Likewise, AI chip sales are soaring, and more powerful and specialized AI chips are being steadily introduced. At this writing, Nvidia, the leading AI chip provider is now the third-most-valuable company in the world, valued at over $2.2 trillion. Nvidia is both developing new AI chips and acquiring smaller AI companies that design processors and develop AI applications. [1]

Other chip companies are also thriving. Micron Technology is reporting record sales, much of them from supplying memory chips to Nvidia. AMD provides chips that rival Nvidia’s flagship AI machine learning chip. And Intel is getting $8.5 billion from a US federal program (CHIPS) to support its goal to build the largest AI chip manufacturing site in the world. [2,3]

Innovative Chips Bring High Heat

AI chip technology, which evolved from the graphics processing units, GPUs, developed for the data needs of video games, may be the most understood part of the AI world. But this evolution is remarkable. The Nvidia A100 AI Chip features 54 billion transistors. By comparison, an AMD Ryzen 7 1700 gaming processor for a contemporary PC has 4.8 million transistors. [4]

By leveraging parallel processing capabilities, AI chips effectively handle large datasets, allowing multiple tasks to be executed simultaneously. These chips interact with special ASICs, FPGAs, TPUs and VPUs to perform machine learning and neural network processing. The AI networks can solve complex algorithms and are teaching computers to process data in a way that is inspired by the human brain.

For example, AI can use inference – combining reasoning and decision-making based on available information – to apply real-world knowledge for facial recognition, gesture identification, natural language interpretation, image searching  and much more. [5]

AI chips demand high power to support increased processing demand. As a result, excessive waste heat can degrade performance or trigger system failure. AI system designers depend on thermal management solutions to manage AI processor temperatures. Cooling resources at both chip-level and facility, e.g. data center scale are needed to keep AI chips functioning at proper temperatures.

Liquid Cooling AI Chips

The heavy lifting in AI processing is done in data centers, which are the focus of most technical developments. Their high concentration of high-power chips presents formidable heat management challenges, especially when the thermal design power of the GPU has increased over the past two decades, rising from 150 watts to more than 700 watts.

Now consider the recently unveiled 1200 watt Nvidia Blackwell B200 tensor core chip—the company’s most powerful single-chip GPU, with 208 billion transistors—which Nvidia says can reduce AI inference operating costs (such as running ChatGPT) and energy. Two of these B200 chips are combined with an Nvidia Grace CPU to complete the newly released, even higher-performing GB200. Its total projected power draw: up to 2,700 watts.

The GB200 chip is a key part of Nvidia’s new GB200 NVL72, a liquid-cooled data center computer system designed specifically for AI training and inference tasks. Amazon Web Services, Dell Technologies, Google, Meta, Microsoft, OpenAI, Oracle, Tesla, and xAI, are expected to adopt the Blackwell platform. [8]

The ever increasing number of transistors attached to data center PCBs translates to higher performance but also more heat than ever before. Liquid cooling systems, like in the Nvidia data center system can significantly reduce energy consumption. This leads to lower operating expenses in the long run. It also produces less noise and, for direct on chip cooling, takes up less space. 

Direct to chip or node cooling involves circulating a coolant directly over heat-generating components, including AI chips. This method significantly increases cooling efficiency by removing heat directly at the source. These systems can use a variety of coolants, including water, dielectric fluids, or refrigerants, depending on the application’s needs and the desired cooling capacity. [8]

Immersion cooling takes liquid cooling a step further by submerging the entire server, or parts of it, in a non-conductive liquid. This technique can be highly efficient as it ensures even and thorough heat absorption from all components. Immersion cooling is particularly beneficial for high-performance computing (HPC) and can dramatically reduce the space and energy required for cooling.

Air Cooling AI Chips

Nvidia’s Jetson chips bring accelerated AI performance to IoT and Edge applications in a power-efficient and compact form factor (smaller than 100mm x 100mm). Less power-consuming (up to 75 watts) than data center AI chips, thermal management is still needed. Jeston components are typically cooled with heat sinks, which can be configured as active (with attached fan) or passive (fanless). 

Conclusion

Artificial intelligence is hot in the marketplace. So are AI chips. As complex as they are, simply surpassing a heat threshold can affect their proper function. Their thermal management is essential.