Tag Archives: data center cooling

Cooling AI Data Centers

How important are AI data centers? In just months, Elon Musk’s xAI team converted a factory outside Memphis into a cutting-edge, 100,000-GPU center for training the Colossus supercomputer—home to the Grok chatbot.

Initially powered by temporary gas turbines (later replaced by grid power), Colossus installed its first 100,000 chips in only 19 days, drawing praise from NVIDIA CEO Jensen Huang. Today, it operates 200,000 GPUs, with plans to reach 1 million GPUs by the end of 2025. [1]

Figure 1 – Elon Musk’s 1 Million Sq Ft xAI Colossus Supercomputer Facility near Memphis, TN. [1]

There are about 12,000 data centers throughout the world, nearly half of them in the United States. Now, more and more of these are being built or retrofitted for AI-specific workloads. Leaders include Musk’s xAI, Microsoft, Meta, Google, Amazon, OpenAI, and others.

High power is essential for such operations, and like computational electronics of all sizes heat issues need to be resolved.

GenAI

A key driver of data center growth is Generative AI (GenAI)—AI that creates text, images, audio, video, and code using deep learning. Chatbots and large language model ChatGPT are examples of GenAI, along with text-to-image models that generate images from written descriptions.

Managing all this is possible from new generations of processors, mainly GPUs. They all draw on higher levels of power and generate higher amounts of heat.

Figure 2 – Advanced AI Processor, the NVIDIA GH200 Grace Hopper Superchip with Integrated CPU to Increase Speed and Performance. [2,3]

AI data centers prioritize HPC hardware: GPUs, FPGAs, ASICs, and ultra-fast networking. Compared to CPUs (150–200 W), today’s AI GPUs often run >1,000 W.  . To handle massive datasets and complex computations in real-time they need significant power and cooling infrastructure.

Data Center Cooling Basics

Traditional HVAC was sufficient for older CPU-driven data centers. Today’s AI GPUs demand far more cooling, both at the chip level and facility-wide. This has propelled a need for more efficient thermal management systems at both the micro (server board and chip) and macro (server rack and facility) levels. [4]

Figure 3 – The Colossus AI Supercomputer Now Runs 200,000 GPUs. It Operates at 150MW Power, Equivalent to 80,000 Households. [5]

At Colossus, Supermicro 4U servers house NVIDIA Hopper GPUs cooled by:

  • Cold plates
  • Coolant distribution manifolds (1U between each server)
  • Coolant distribution units (CDUs) with redundant pumps at each rack base [6]

Each 4U server is equipped with eight NVIDIA H100 Tensor Core GPUs. Each rack contains eight 4U servers, totaling 64 GPUs per rack.

Between every server is a 1U manifold for liquid cooling. They connect with CDUs, heat-exchanging Coolant Distribution Units at the bottom of each rack that include a redundant pumping system. The choice of coolant is determined by a range of hardware and environmental factors.

Figure 4 – Each Colossus Rack Contains Eight 4U Servers, Totaling 64 GPUs Per Rack. Between Each Server is a 1U Manifold for Liquid Cooling. [7]
Figure 5 – The Base of Each Rack Has a 4U CDU Pumping System with Redundant Liquid Cooling. [7]

Role of Cooling Fans

Fans remain essential for DIMMs, power supplies, controllers, and NICs.

Figure 6 – Rear Door Liquid-Cooled Heat Exchangers. [7]

At Colossus, fans in the servers pull cooler air from the front of the rack, and exhaust the air at the rear of the server. From there, the air is pulled through rear door heat exchangers. The heat exchangers pass warm air through a liquid-cooled, finned heat exchanger/radiator, lowering its temperature before it exits the rack.

Direct-to-Chip Cooling

NVIDIA’s DGX H100 and H200 server systems feature eight GPUs and two CPUs that must run between 5°C and 30°C. An AI data center with a high rack density houses thousands of these systems performing HPC tasks at maximum load. Direct liquid cooling solutions are required.

Figure 7 – An NVIDIA DGX H100/H200 System Featuring Eight GPUs [8]
Figure 8 – The NVIDIA H100 SmartPlate Connects to a Liquid Cooling System to Bring Microconvective Chip-Level Cooling That Outperforms Air Cooling by 82%. [9]

Direct liquid cooling (cold plates contacting the GPU die) is the most effective method—outperforming air cooling by 82%. It is preferred for high-density deployments of the H100 or GH200.

Scalable Cooling Modules

Colossus represents the world’s largest liquid-cooled AI cluster, using NVIDIA + Supermicro technology. For smaller AI data centers, Cooling Distribution Modules (CDMs) provide a compact, self-contained solution.

Figure 9 – The iCDM-X Cooling Distribution Module from ATS Includes Pumps, Heat Exchanger and Liquid Coolant for Managing Heat from AI GPUs and Other Components. [10]

Most AI data centers are smaller, and power and cooling needs are lower, but essential. Many heat issues can be resolved using self-contained Cooling Distribution Modules.

The compact iCDM-X cooling distribution module provides up to 1.6MW of cooling for a wide range of AI GPUs and other chips. The module measures and logs all important liquid cooling parameters. It uses using just 3kW of power, and no external coolant is required.

These modules include:

•         Pumps

•         Heat exchangers

•         Cold plates

•         Digital monitoring (temp, pressure, flow)

Their sole external component is one or more cold plates removing heat from AI chips. ATS provides an industry-leading selection of custom and standard cold plates, including the high-performing ICEcrystal series.

Figure 10 – The ICEcrystal Cold Plates Series from ATS Provide 1.5 kW of Jet Impingement Liquid Cooling Directly onto AI Chip Hotspots.

Cooling Edge AI and Embedded Applications

AI isn’t just for big data centers—edge AI, robotics, and embedded systems (e.g., NVIDIA Jetson Orin, AMD Kria K26) use processors running under 100 W. These are effectively cooled with heat sinks and fan sinks from suppliers like Advanced Thermal Solutions. [11]

Figure 11 – High Performance Heat Sinks for NVIDIA and AMD AI Processors in Embedded and Edge Applications. [11]

NVIDIA also partners with Lenovo, whose 6th-gen Neptune cooling system enables full liquid cooling (fanless) across its ThinkSystem SC777 V4 servers—targeting enterprise deployments with NVIDIA Blackwell + GB200 GPUs. [12]

Figure 12 – Lenovo’s Neptune Direct Water Cooling Removes Heat from Power Supplies, for Completely Fanless Operation. [12]

Benefits gained from the Neptune system include:

  • Full system cooling (GPUs, CPUs, memory, I/O, storage, regulators)
  • Efficient for 10-trillion-parameter models
  • Improved performance, energy efficiency, and reliability

Conclusion

With surging demand, AI data centers are now a major construction focus. Historically, cooling problems are the #2 cause of data center downtime (behind power issues). With the high power needed for AI computing, these builds should carefully fit with their local communities in terms of electrical needs and sources, and water consumption. [13]

AI workloads will increase U.S. data center power demand by 165% by 2030 (Goldman Sachs), with nearly double 2022 levels (IBM/Newmark). Sustainable design and resource-conscious cooling are essential for the next wave of AI infrastructure. [14,15]

References

1. The Guardian, https://www.theguardian.com/technology/2025/apr/24/elon-musk-xai-memphis

2. Fibermall, https://www.fibermall.com/blog/gh200-nvidia.htm

3. NVIDA, https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip?ncid=no-ncid

4. ID Tech Ex, https://www.idtechex.com/en/research-report/thermal-management-for-data-centers-2025-2035-technologies-markets-and-opportunities/1036

5. Data Center Frontier, https://www.datacenterfrontier.com/machine-learning/article/55244139/the-colossus-ai-supercomputer-elon-musks-drive-toward-data-center-ai-technology-domination

6. Supermicro, https://learn-more.supermicro.com/data-center-stories/how-supermicro-built-the-xai-colossus-supercomputer

7. Serve The Home, https://www.servethehome.com/inside-100000-nvidia-gpu-xai-colossus-cluster-supermicro-helped-build-for-elon-musk/2/

8. Naddod, https://www.naddod.com/blog/introduction-to-nvidia-dgx-h100-h200-system

9. Flex, https://flex.com/resources/flex-and-jetcool-partner-to-develop-liquid-cooling-ready-servers-for-ai-and-high-density-workloads

10. Advanced Thermal Solutions, https://www.qats.com/Products/Liquid-Cooling/iCDM

11. Advanced Thermal Solutions, https://www.qats.com/Heat-Sinks/Device-Specific-Freescale

12. Lenovo, https://www.lenovo.com/us/en/servers-storage/neptune/?orgRef=https%253A%252F%252Fwww.google.com%252F

13. Deloitte, https://www2.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2025/genai-power-consumption-creates-need-for-more-sustainable-data-centers.html

14.GoldmanSachs, https://www.goldmansachs.com/insights/articles/ai-to-drive-165-increase-in-data-center-power-demand-by-2030

15. Newmark, https://www.nmrk.com/insights/market-report/2023-u-s-data-center-market-overview-market-clusters

October Webinar 2021 Cooling Methodologies for High Power Systems and Components, 10-21-21 2PM EST

High power systems of various kinds have become even more widley deployed than in the past. From 15kW IGBT power systems, to 50kW high density power converters, to even higher wattage for computing racks. As the need for applications for systems with this power has grown, so too has the thermal management solutions that allows these systems to operate at their intended performance levels.

This webinar covers the newest strategies and technologies for these very high power applications. In this technology review webinar attendees will learn:

  • How the integration of the thermal architecture with electronic architecture can can significantly improve the effectiveness of thermal management.
  • The use of passive two phase thermosyphon loops for cooling data center racks
  • How to passively cool outdoor 7.2kW power supply electronics enclosures
  • The application of a closed rack cooling system based on pulsating heat pipes
  • The development of a loop heat pipe with kW-class heat transport capability
  • A new class of graphene enhanced heat pipe with heat dissipation characteristics 3.5x higher than copper
  • Using multi-stage heat pipe loops for cooling high heat density data centers
  • The application of pin finned cold plates for cooling high power density converters (HPDC)
  • Many more new and latest developments in thermal management for high power systems and components will be reviewed. This webinar, presented by thermal management expert Dr. Kaveh Azar, Ph.D., will address all these issues and more. The webinar is one hour in length with time for questions and answers afterwards online and after the webinar concludes.

Technology Review: Data Center Cooling

(This article was featured in an issue of Qpedia Thermal e-Magazine, an online publication dedicated to the thermal management of electronics. To get the current issue or to look through the archives, visit http://www.qats.com/Qpedia-Thermal-eMagazine.)

Qpedia continues its review of technologies developed for electronics cooling applications. We are presenting selected patents that were awarded to developers around the world to address cooling challenges. After reading the series, you will be more aware of both the historic developments and the latest breakthroughs in both product design and applications.

Data Center Cooling

This article explores recent patents and technical advancements from the data center cooling industry. (Wikimedia Commons)

We are specifically focusing on patented technologies to show the breadth of development in thermal management product sectors. Please note that there are many patents within these areas. Limited by article space, we are presenting a small number to offer a representation of the entire field. You are encouraged to do your own patent investigation.

Further, if you have been awarded a patent and would like to have it included in these reviews, please send us your patent number or patent application.

In this issue our spotlight is on data center cooling solutions.

There are many U.S. patents in this area of technology, and those presented here are among the more recent. These patents show some of the salient features that are the focus of different inventors.

Hybrid Immersion Cooled Server with Integral Spot and Bath Cooling

US 7724524 B1, Campbell, L., Chu, R., Ellsworth, M., Iyengar, M. and Simons, R.

An immersion cooling apparatus and method is provided for cooling of electronic components housed in a computing environment. The components are divided into primary and secondary heat generating components and are housed in a liquid sealed enclosure. The primary heat generating components are cooled by indirect liquid cooling provided by at least one cold plate having fins. The cold plate is coupled to a first coolant conduit that circulates a first coolant in the enclosure and supplies the cold plate. Immersion cooling is provided for secondary heat generating components through a second coolant that will be disposed inside the enclosure such as to partially submerge the cold plate and the first coolant conduit as swell as the heat generating components.

The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method and associated hybrid immersion cooling apparatus for cooling of electronic components housed in a computing environment. The components are categorized as primary and secondary heat generating components and are housed in a liquid tight enclosure. The primary heat generating components are cooled by indirect liquid cooling provided by at least one cold plate having fins.

The cold plate is coupled to a first coolant conduit that circulates a first coolant in the enclosure and supplies the cold plate. Immersion or direct liquid cooling is provided for secondary heat generating components through a second coolant that will be disposed inside the enclosure such as to partially submerge the cold plate and the first coolant conduit as Well as the heat generating components.

In one embodiment, the cold plate and the coolant conduit each comprise extended external surfaces used to transfer heat from the second coolant to the first coolant. In one embodiment the second coolant cools the electronics via free convection or a combination of free convection and boiling while an alternate embodiment provides a sub merged pump to aid circulation. In yet another embodiment, the electronics are housed on an electronics board that is tilted at an angle to also aid circulation of the second coolant.

Thermal Caching For Liquid Cooled Computer Systems

US 7436666 B1, Konshak, M.

The present invention involves supplementing the liquid cooling path within a given rack of electronic components in order to reduce the temperature rise of critical components before damage or data loss can occur in the event of a primary cooling system failure. Liquid cooled racks generally have piping and heat exchangers that contain a certain volume of liquid or vapor that is being constantly replaced by a data center pump system. Upon a data center power or pump failure, the How stops or is diminished. The coolant, however, is still present within the rack components. Upon detection of a lack of How, an unexpected rise in the coolant temperature, or a significant reduction in How pressure, and before any liquid can be drained away from the racks, an electrically or hydraulically operated valve bypasses the supply and return of the data center cooling path, creating an independent closed loop supplemental cooling system within the rack itself.

One embodiment of a system for mitigating a failure of a data center’s liquid cooling system is shown. A high level block diagram of one embodiment of a system for thermal caching in a liquid cooled computer system is presented. Under normal circumstances, coolant is delivered to a rack housing a plurality of liquid cooled components by a data center liquid cooling system supply and return lines. The supply and return lines are coupled to the rack by quick disconnecting and the coolant is pumped through the conduits by one or more data center pumps. Within the rack a secondary cooling loop is established that is in fluid communication with the data center cooling loop.

Upon entering the rack the data center supply line enters a monitoring device. In one embodiment of the present invention the device monitors fluid flow, pressure, and/or temperature. Other characteristics of the fluid that can also be used to identify a failure of the data center cooling system as imposed on the supply line. Thereafter, and before allowing the coolant to access any of the electronic components housed within the rack, a one way valve is placed on the supply lid entering the rock. The valve allows fluid to pass from the data center cooling supply line into the rack but prevents flow from regressing toward the supply line should the flow stop and/or an adverse pressure gradient is experienced.

Similarly, a two-way valve is placed on the coolant return line that returns heated coolant from the electronic components to the data center coolant return line. The two way valve is in communication with the monitoring device and capable of receiving a signal that indicates a failure in the primary or data center liquid cooling system.

Upon receiving such a signal from the monitoring device, the two-way valve closes either electrically or hydraulically and diverts the return coolant from the electronic devices to the supplemental or secondary liquid cooling loop. The supplemental liquid cooling loop, which is housed entirely within the rack, thereafter operates independent of the data center liquid cooling system.

Modular High-Density Computer System

US 7688578 B2, Mann, R., Landrum, G. and Hintz, R.

A modular high-density computer system has an infrastructure that includes a framework component forming a plurality of bays and has one or more cooling components. The computer system also has one or more computational components that include a rack assembly and a plurality of servers installed in the rack assembly. Each of the one or more computational components is assembled and shipped to an installation site separately from the infrastructure and then installed within one of the plurality of bays after the infrastructure is installed at the site.

Historically, a room oriented cooling infrastructure was sufficient to handle this cooling problem. Such an infrastructure was typically made up of one or more bulk air conditioning units designed to cool the room to some average temperature. This type of cooling infrastructure evolved based on the assumption that the computer equipment is relatively homogeneous and that the power density is on the order 1 -2 Kilowatts per rack. The high density racks mentioned above, however, can be on the order of 20 Kilowatts per rack and above. This increase in power density, and the fact that the equipment can be quite heterogeneous leads to the creation of hot spots that can no longer be sufficiently cooled with simple room-oriented cooling systems.

By separating the computational components of a high density computer system from the requisite cooling, power distribution, data l/O distribution, and housing infrastructure components, significant advantages can be achieved by embodiments of the invention over current configurations of such computer systems. The computational components can be assembled and tested for proper operation while the infrastructure components are shipped separately in advance of the computational components. Thus, the infrastructure cooling components, data and power distribution components, and framework components can be delivered and their time-intensive installation completed more easily and prior to receiving the tested computational components.

Moreover, the shipping of the computational components separately from the infrastructure components achieves less laborious and less costly delivery and handling of the components. Finally, it makes it easier for the data center to upgrade or add additional infrastructure and infrastructure components to a current installation.

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.

Holiday Sale: 25 % Off Qpedia Book Series!

 

coolingZONE is running a holiday promotion on the Qpedia Book Series, discounting 25% off the normal retail value until December 31, 2012. Qpedia Thermal eMagazine is the official media sponsor of coolingZONE. The Qpedia book series provides an expert resource for engineering professionals, professors, students and others who want to learn more about the theories and applications of electronics thermal management.

The four volume set contains nearly 200 in-depth articles, researched and written by veteran engineers. They address the most critical areas of electronics cooling, with a wide spectrum of topics and thorough technical explanation. Each article features color illustrations and images along with important thermal calculations and formulas.

Qpedia Volume 1 topic highlights:

– Heat sink design

– Pool boiling

– Sensors

– Thermal chassis

– Pitot tubes

– Heat transfer

Qpedia Volume 2 topic highlights:

– Thermal grease

– Vapor chamber cooling

– Liquid cooling

– Natural convection cooling

– Phase change materials

Qpedia Volume 3 topic highlights:

– CFD modeling

– PCB optimization

– Jet impingement cooling

– LED cooling

– Cold plates

– Refrigeration systems

Qpedia Volume 4 topics highlights:

– Multilayer mini-channel heat sinks

– Electro-osmotic pumps

– Thermoelectric coolers

– High altitude heat transfer

– Data center cooling

– Defense electronics

– Automotive electronics cooling

Check out the video on Qpedia Volume 4

Over fifteen thermal and mechanical engineers combined their expertise to create the articles in this useful collection, including Kaveh Azar, Ph.D. and Bahman Tavassoli, Ph.D., both of whom are internationally recognized experts in the field of the thermal management of electronics. Qpedia Thermal eMagazine, published by Advanced Thermal Solutions, Inc. (ATS) was launched in 2007 as a technical publication dedicated to the thermal management of electronics.

Filled with careful explanations, step-by-step calculations, and practical examples from real life challenges, this expert collection of articles is a must have for any mechanical, electrical, and thermal engineer or anyone interested in the rapidly growing field of electronics cooling.

Order the Qpedia Book Series with 25% off now at www.coolingzone.com The books can be purchased as a complete set, combination, or individually. Volume discounts for organizations and academic institutions are also available.