Category Archives: thermal management

Hybrid Cooling: Air and Liquid Double-Team Heat in Data Centers

Posted on September 8, 2026 by | Leave a comment

Much of the focus on data centers is on their very high power and cooling demands and the impact those demands have on local electrical grids and water supplies. Water is needed for thermally managing the hot-running processors, especially GPUs, at the heart of AI data center computing. Many other essential components also require cooling, but at lower levels, allowing them to be cooled using forced air.

This is why hybrid cooling, using both liquid and air methods, is an effective way to keep hot data center components within their operating temperature ranges. Hybrid cooling enables a more balanced, incremental approach to managing different types of heat loads efficiently. Liquid cooling handles concentrated heat loads, while airflow manages the remaining thermal load.

Figure 1 – Hybrid Air and Liquid Cooling Systems on a Data Center Server [2]

Forced air cools lower power components using heat sinks and air from fans and blowers. But traditional air cooling becomes inefficient as board and component densities climb, and power levels increase, justifying the need for high capacity air cooling that is not necessarily a larger fan of heat sink.  

Liquid cooling is provided by continuously circulating liquid coolant across the hot modules, drawing their heat away. While air cooling is provided by rows of fans, liquid cooling is a complex system of tubing and connectors, coolant distribution units (CDUs), cold plates, and coolants. [3]

Data Center Applications

Let’s look at hybrid cooling inside a typical server in an AI data center, where it can be one of thousands of servers running in tightly and precisely arranged rows of cabinets. Both air and liquid cooling play essential roles in thermally managing the many powered components.

Air Cooling

Server cases are explicitly designed to function as wind tunnels. Server airflow is generated by internal fans that create air pressure. Typically, these are high-static-pressure, counter-rotating fans that produce cooling airflow passing over and around motherboards and hot components. Fan pressure pushes heated air out of the server while drawing cool air in from outside. [4]

At the component level, air-cooling designs rely on conduction to move heat from a chip to a metal heat sink, followed by forced convection driven by high-static-pressure fans to carry the heat away.

Figure 2 – Fans Pull Air In and Across the Board. Warmed Air Exhausts Out the Back, e.g., into a Hot Aisle. [4]

Some commonly air-cooled components include:

  • Memory (DIMMs / DDR5): Cooled via linear front-to-back airflow channels passing across the motherboard.
  • Networking (NICs / SmartNICs / OCP 3.0 cards): Low-to-moderate-power PCIe or OCP networking modules.
  • Storage (NVMe / SSDs): Front-accessible U.2/U.3 or E1.S drives.
  • Power Supply Units (PSUs): Integrated hot-swap power supply modules equipped with their own independent, self-contained fans.
  • CPUs: Some lower-power CPUs can be air-cooled using low-profile fan-sinks or vapor-chamber heat sinks. [5]

In standard 1U and 2U multi-node enterprise server racks, CPUs and other components are almost always cooled passively using aluminum or copper block heat sinks that rely entirely on the high-RPM chassis fans located at the front of the server case.

Despite its broad applicability, air cooling is typically used for less than 20 kilowatts (kW) per rack in conventional designs, although it can reach 35 kW in exceptional cases. Anything beyond that typically requires liquid cooling. [6] Of course, the selection of a cooling system is governed by device junction temperature rather than total wattage.

Liquid Cooling

Liquid cooling systems circulate coolant through a network of tubing, absorbing heat from hot components, most notably high-power GPUs, many of which operate at 1,000 W.

Figure 3 – Liquid Cooling on a Data Center GPU. [7]

In data center servers, liquid cooling primarily targets ultra-high-heat-producing processors, specifically GPUs and CPUs, using direct-to-chip cold plates. In some advanced high-density 1U designs, liquid loops also extend coverage to adjacent high-power components.

Some commonly liquid-cooled components include:

  • GPUs / Accelerators: The core AI training and inference engines that generate the vast majority of server heat.
  • CPUs: Host processors that manage system instructions and data pipeline feeds to accelerators.
  • Network/AI Switches & Co-processors: Specialized fabric chips, such as NVSwitches, mounted on the board to link multiple processors at high speeds. [8]

Figure 4 – A Hybrid-Cooled 1U Rack Mount Server Chassis by Titan Rig includes 490W Cooling Capacity. A Powerfan Hub Bundle Provides Air-Cooling for Lower Power Components. [9]

Power of Two

Hybrid cars combine gas engines and electric motors to improve fuel efficiency. Similarly, hybrid cooling in data center servers combines air and liquid systems for greater thermal-management efficiency. Air cooling manages ambient and lower-density heat loads, typically up to 35 kW per server board. Liquid cooling targets high-intensity heat directly at the component level, such as CPUs and GPUs. Both systems operate simultaneously, reducing the thermal burden placed on either system alone.

Figure 5 – Liquid Cooling from ZutaCore Closely Neighbors an Air Cooling Heat Sink on a Server Board. [3]

Data center cooling isn’t only about installing fans and circulating coolant. Strategic design combined with optimization is crucial for ensuring that data center infrastructure and components operate at their best. This approach not only prevents devices from overheating but also maximizes the efficient use of data center resources, including power and cooling systems.

While this post focuses on component cooling within servers, data centers also have other thermal-management systems. In particular, their HVAC systems deliver chilled airflow to targeted areas within the facility. By optimizing airflow through vents and barriers, data centers can minimize areas of stagnant air. Configuring server racks in alternating hot- and cold-aisle rows maximizes efficiency by ensuring that cooled air is delivered where it is needed most while preventing hot and cold air from mixing.

References

  1. Top Image https://www.gigabyte.com/Enterprise/GPU-Server/G262-ZL0-rev-G00
  2. Tech Stories, https://www.techstories.co/liquid-cooling-leak-destroys-millions-of-dollars-in-gpus/
  3. Chatsworth, https://www.chatsworth.com/en-us/resources/blogs/2026/what-is-hybrid-cooling-a-smarter-approach-to-data-center-thermal-design/
  4. EziBlank, https://eziblank.com/blog/2022/11/29/how-airflow-works-inside-of-a-server/
  5. Cooltron, https://www.cooltron.com/news/thermal-engineer’s-guide-to-choosing-ai-server-cooling-fans-2078026334647910402
  6. Digital Reality, https://www.digitalrealty.com/resources/blog/future-of-data-center-cooling
  7. Jetcool, https://jetcool.com/post/what-is-direct-liquid-cooling-for-ai-data-centers/
  8. Solidigm, https://www.solidigm.com/products/technology/liquid-cooling-basics.html
  9. Titan Rig, https://www.titanrig.com/1u-liquid-cooled-rack-mount-server-chassis-with-490w-cooling-capacity-xpx-pro-cpu-water-block-es-guardian-powerfan-hub-bundle.html

Leave a comment

Posted in Air Cooling, cooling, Liquid Cooling, Thermal Design, thermal management

Tagged , , ,

Accurate Instruments are Essential When Measuring for Thermal Management Solutions

Posted on August 14, 2026 by | Leave a comment

By collecting and analyzing more relevant measurement data, electronic cooling systems can be designed for improved performance, reliability, and cost. It’s essential these measurements be as accurate as possible. Critical data points include component temperature, air temperature, air flow rate, and often air pressure. Accuracy becomes even more important when performing accelerated testing as seen in this post.

Research-grade instruments and specialized sensors, properly calibrated and operated, can measure chip temperatures and surrounding elements, such as local airflow. Separate instruments can be used to quantify specific conditions, but more real-use data is obtained when all points are collected simultaneously. Taken over a range of time and conditions, simultaneous measurements provide more accurate, actionable information. They also reduce the cost and set-up time for multiple studies.

Fig 1 – Input Ports for a Laboratory Instrument that Simultaneously Measures Air Velocity, Temperature and Pressure. [ATS]
Fig 2 – High-quality Thermal Data Reduces Uncertainties in Cooling Solutions than Calculated Data. [ATS]

Accelerated Product Reliability Testing

Reliability testing at different temperatures and conditions is commonly performed to identify expected product life. When an electronic system is running, it continuously experiences temperature and power cycling. These fluctuations are typically normal, but should be factored into a comprehensive thermal profile study.

It’s not practical to wait years to see if a component fails. Accelerated life testing provides faster predictions. A commonly used predictive tool is the Arrhenius model shown below.

Accelerated life testing using the Arrhenius model is a method for predicting how long a product will last under normal conditions by running it at elevated temperatures for a shorter time. The temperature increase speeds up the same failure processes that would eventually occur in the field—compressing years of life into weeks of testing.

Accelerated life testing Arrhenius calculators can help estimate field lifetime, test duration, acceleration factors, activation energy, and field failure rates. These tools are useful when real-time qualification can’t be done. This is typical with electronic components and devices that need fast market introduction. One such calculator is found here: Accelerated Life Test Arrhenius Calculator. [1]

The Arrhenius Model handles pure thermal stress, while the Eyring Model includes temperature plus other active forces like humidity or voltage. [1,2]

Highly Accurate Thermal Instruments

Besides its design, a thermal instrument’s construction and calibration should be high quality, and not a source of more errors. Accurate, research-grade instruments are essential for valid engineering practices. Without them, reliability predictions for a new product are actually less reliable themselves.

Fig 3 – Candlestick Sensors are Thin Profile (0.5 mm) with a Small Base (9.5 mm dia) for Collecting Air Temperature and Velocity Data at Single or Multiple Locations. [ATS]
Fig 4 – Here, Air Temperature and Velocity Measurements are Taken Every Second by an eATVS-8 Instrument. stageVIEW Software Provides a Wide Range of Data Collection Choices. [ATS]



Here are some of the ATS instruments available for temperature and velocity measurement and providing the highest-quality thermal reliability test results.

Conclusion

Temperature has a significant effect on reaction rate and the subsequent reliability prediction. Small errors in temperature and airflow measurements can significantly impact reliability predictions. Such errors typically occur in the measurement process or from using inaccurate instruments.

If thermal data is inaccurate, outcomes can include over-designed systems, cost overruns, and products not shipping.

Laboratory-quality thermal instruments can reduce or eliminate measurement and calculation mistakes, leading to higher quality, faster to market and ultimately lower cost products. Just as important, collected data needs to be correctly interpreted. Expert thermal engineers can do just that with data provided or that they obtain themselves.

More information on these topics can be seen at these ATS videos:

How to Properly Measure Air Velocity in Electronic Systems [Link] [3]

How to Perform and Understand Temperature Measurement within Electronic Systems [Link] [4]

stageVIEW Software Demo [Link] [5]

References

  1. Firgelli Automations, https://www.firgelliauto.com/blogs/calculators/accelerated-life-test-arrhenius-calculator
  2. NIST, https://www.itl.nist.gov/div898/handbook/apr/section1/apr152.htm
  3. Advanced Thermal Solutions, Inc., https://vimeo.com/reviews/a4e9a72d-d228-489f-8edb-a98e440a8ef1/videos/113427506
  4. Advanced Thermal Solutions, Inc., https://vimeo.com/reviews/63784f1b-a32d-4f90-86b1-5bf362715619/videos/113432468
  5. Advanced Thermal Solutions, Inc., https://www.youtube.com/watch?v=Lx5NlAI5TuM

Leave a comment

Posted in Analysis, Consulting, Engineering, Instrument, Sensor, Thermal Design, thermal engineering, thermal management

Tagged , , ,

Cooling AI Data Centers

Posted on June 27, 2025 by | 1 comment

How important are AI data centers? In just months, Elon Musk’s xAI team converted a factory outside Memphis into a cutting-edge, 100,000-GPU center for training the Colossus supercomputer—home to the Grok chatbot.

Initially powered by temporary gas turbines (later replaced by grid power), Colossus installed its first 100,000 chips in only 19 days, drawing praise from NVIDIA CEO Jensen Huang. Today, it operates 200,000 GPUs, with plans to reach 1 million GPUs by the end of 2025. [1]

Figure 1 – Elon Musk’s 1 Million Sq Ft xAI Colossus Supercomputer Facility near Memphis, TN. [1]

There are about 12,000 data centers throughout the world, nearly half of them in the United States. Now, more and more of these are being built or retrofitted for AI-specific workloads. Leaders include Musk’s xAI, Microsoft, Meta, Google, Amazon, OpenAI, and others.

High power is essential for such operations, and like computational electronics of all sizes heat issues need to be resolved.

GenAI

A key driver of data center growth is Generative AI (GenAI)—AI that creates text, images, audio, video, and code using deep learning. Chatbots and large language model ChatGPT are examples of GenAI, along with text-to-image models that generate images from written descriptions.

Managing all this is possible from new generations of processors, mainly GPUs. They all draw on higher levels of power and generate higher amounts of heat.

Figure 2 – Advanced AI Processor, the NVIDIA GH200 Grace Hopper Superchip with Integrated CPU to Increase Speed and Performance. [2,3]

AI data centers prioritize HPC hardware: GPUs, FPGAs, ASICs, and ultra-fast networking. Compared to CPUs (150–200 W), today’s AI GPUs often run >1,000 W.  . To handle massive datasets and complex computations in real-time they need significant power and cooling infrastructure.

Data Center Cooling Basics

Traditional HVAC was sufficient for older CPU-driven data centers. Today’s AI GPUs demand far more cooling, both at the chip level and facility-wide. This has propelled a need for more efficient thermal management systems at both the micro (server board and chip) and macro (server rack and facility) levels. [4]

Figure 3 – The Colossus AI Supercomputer Now Runs 200,000 GPUs. It Operates at 150MW Power, Equivalent to 80,000 Households. [5]

At Colossus, Supermicro 4U servers house NVIDIA Hopper GPUs cooled by:

  • Cold plates
  • Coolant distribution manifolds (1U between each server)
  • Coolant distribution units (CDUs) with redundant pumps at each rack base [6]

Each 4U server is equipped with eight NVIDIA H100 Tensor Core GPUs. Each rack contains eight 4U servers, totaling 64 GPUs per rack.

Between every server is a 1U manifold for liquid cooling. They connect with CDUs, heat-exchanging Coolant Distribution Units at the bottom of each rack that include a redundant pumping system. The choice of coolant is determined by a range of hardware and environmental factors.

Figure 4 – Each Colossus Rack Contains Eight 4U Servers, Totaling 64 GPUs Per Rack. Between Each Server is a 1U Manifold for Liquid Cooling. [7]
Figure 5 – The Base of Each Rack Has a 4U CDU Pumping System with Redundant Liquid Cooling. [7]

Role of Cooling Fans

Fans remain essential for DIMMs, power supplies, controllers, and NICs.

Figure 6 – Rear Door Liquid-Cooled Heat Exchangers. [7]

At Colossus, fans in the servers pull cooler air from the front of the rack, and exhaust the air at the rear of the server. From there, the air is pulled through rear door heat exchangers. The heat exchangers pass warm air through a liquid-cooled, finned heat exchanger/radiator, lowering its temperature before it exits the rack.

Direct-to-Chip Cooling

NVIDIA’s DGX H100 and H200 server systems feature eight GPUs and two CPUs that must run between 5°C and 30°C. An AI data center with a high rack density houses thousands of these systems performing HPC tasks at maximum load. Direct liquid cooling solutions are required.

Figure 7 – An NVIDIA DGX H100/H200 System Featuring Eight GPUs [8]
Figure 8 – The NVIDIA H100 SmartPlate Connects to a Liquid Cooling System to Bring Microconvective Chip-Level Cooling That Outperforms Air Cooling by 82%. [9]

Direct liquid cooling (cold plates contacting the GPU die) is the most effective method—outperforming air cooling by 82%. It is preferred for high-density deployments of the H100 or GH200.

Scalable Cooling Modules

Colossus represents the world’s largest liquid-cooled AI cluster, using NVIDIA + Supermicro technology. For smaller AI data centers, Cooling Distribution Modules (CDMs) provide a compact, self-contained solution.

Figure 9 – The iCDM-X Cooling Distribution Module from ATS Includes Pumps, Heat Exchanger and Liquid Coolant for Managing Heat from AI GPUs and Other Components. [10]

Most AI data centers are smaller, and power and cooling needs are lower, but essential. Many heat issues can be resolved using self-contained Cooling Distribution Modules.

The compact iCDM-X cooling distribution module provides up to 1.6MW of cooling for a wide range of AI GPUs and other chips. The module measures and logs all important liquid cooling parameters. It uses using just 3kW of power, and no external coolant is required.

These modules include:

•         Pumps

•         Heat exchangers

•         Cold plates

•         Digital monitoring (temp, pressure, flow)

Their sole external component is one or more cold plates removing heat from AI chips. ATS provides an industry-leading selection of custom and standard cold plates, including the high-performing ICEcrystal series.

Figure 10 – The ICEcrystal Cold Plates Series from ATS Provide 1.5 kW of Jet Impingement Liquid Cooling Directly onto AI Chip Hotspots.

Cooling Edge AI and Embedded Applications

AI isn’t just for big data centers—edge AI, robotics, and embedded systems (e.g., NVIDIA Jetson Orin, AMD Kria K26) use processors running under 100 W. These are effectively cooled with heat sinks and fan sinks from suppliers like Advanced Thermal Solutions. [11]

Figure 11 – High Performance Heat Sinks for NVIDIA and AMD AI Processors in Embedded and Edge Applications. [11]

NVIDIA also partners with Lenovo, whose 6th-gen Neptune cooling system enables full liquid cooling (fanless) across its ThinkSystem SC777 V4 servers—targeting enterprise deployments with NVIDIA Blackwell + GB200 GPUs. [12]

Figure 12 – Lenovo’s Neptune Direct Water Cooling Removes Heat from Power Supplies, for Completely Fanless Operation. [12]

Benefits gained from the Neptune system include:

  • Full system cooling (GPUs, CPUs, memory, I/O, storage, regulators)
  • Efficient for 10-trillion-parameter models
  • Improved performance, energy efficiency, and reliability

Conclusion

With surging demand, AI data centers are now a major construction focus. Historically, cooling problems are the #2 cause of data center downtime (behind power issues). With the high power needed for AI computing, these builds should carefully fit with their local communities in terms of electrical needs and sources, and water consumption. [13]

AI workloads will increase U.S. data center power demand by 165% by 2030 (Goldman Sachs), with nearly double 2022 levels (IBM/Newmark). Sustainable design and resource-conscious cooling are essential for the next wave of AI infrastructure. [14,15]

References

1. The Guardian, https://www.theguardian.com/technology/2025/apr/24/elon-musk-xai-memphis

2. Fibermall, https://www.fibermall.com/blog/gh200-nvidia.htm

3. NVIDA, https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-superchip?ncid=no-ncid

4. ID Tech Ex, https://www.idtechex.com/en/research-report/thermal-management-for-data-centers-2025-2035-technologies-markets-and-opportunities/1036

5. Data Center Frontier, https://www.datacenterfrontier.com/machine-learning/article/55244139/the-colossus-ai-supercomputer-elon-musks-drive-toward-data-center-ai-technology-domination

6. Supermicro, https://learn-more.supermicro.com/data-center-stories/how-supermicro-built-the-xai-colossus-supercomputer

7. Serve The Home, https://www.servethehome.com/inside-100000-nvidia-gpu-xai-colossus-cluster-supermicro-helped-build-for-elon-musk/2/

8. Naddod, https://www.naddod.com/blog/introduction-to-nvidia-dgx-h100-h200-system

9. Flex, https://flex.com/resources/flex-and-jetcool-partner-to-develop-liquid-cooling-ready-servers-for-ai-and-high-density-workloads

10. Advanced Thermal Solutions, https://www.qats.com/Products/Liquid-Cooling/iCDM

11. Advanced Thermal Solutions, https://www.qats.com/Heat-Sinks/Device-Specific-Freescale

12. Lenovo, https://www.lenovo.com/us/en/servers-storage/neptune/?orgRef=https%253A%252F%252Fwww.google.com%252F

13. Deloitte, https://www2.deloitte.com/us/en/insights/industry/technology/technology-media-and-telecom-predictions/2025/genai-power-consumption-creates-need-for-more-sustainable-data-centers.html

14.GoldmanSachs, https://www.goldmansachs.com/insights/articles/ai-to-drive-165-increase-in-data-center-power-demand-by-2030

15. Newmark, https://www.nmrk.com/insights/market-report/2023-u-s-data-center-market-overview-market-clusters

1 Comment

Posted in Artificial Intelligence, Cabinet, Chassis, Cold Plates, datacenter, Heat Exchangers, Heat Sinks, Jet Impingement, Liquid Cooling, NVIDIA, Thermal Design, thermal management, Uncategorized

Tagged , , , , , , , , , , ,

IoT Electronics and Thermal Management

Posted on February 27, 2025 by | Leave a comment

The IoT – Internet of Things – includes all devices connected to the Internet, a fast growing world, expanding well past PCs and smartphones. IoT devices are our every day and special purpose items like appliances, sensors, and motors.

Per one authority, an IoT product combines hardware and software, measures real-world signals, connects to the Internet, transfers data to a centralized location, and provides value to a customer. [1] There are many such products, already in the tens of billions of IoT connected devices.

Consumer Uses

Among the most common IoT devices are smart speakers, headphones, appliances, and leak detectors. At the enterprise level are smart lighting and security systems in factories, office buildings and public places.

Figure 1. Common IoT Devices. Smart Doorbell Camera Connects Via Mobile App; Wearable Multi-functional Smart Watch; and Smart Water Usage/Leak Detecting Monitor Sends Alerts of Unusual Water Activity. (Kangaroo, StreamLabs, GoodWorkLabs) [2,3,4]

Automotive Use

Secure, strong IoT connection is essential for today’s auto infotainment systems. It enables everything from music platforms to navigation aids to car maintenance and diagnostics. Connected cars have more and more software-reliant components in their cabins, under their hoods and just about everywhere. With IoT-connectivity these components can be updated with OTA (over the air) software fixes without visiting a garage or dealership.

Figure 2. IoT-connected Infotainment System in a Toyota Crown. (PlanetDave) [5]

Industrial Use

The IIoT – Industrial Internet of Things – consists mainly of sensors uploading localized data to monitor and control manufacturing processes. It is active in manufacturing, transportation, and other  areas, from facility energy usage to equipment performance.  IIoT devices collect, analyze, and share data with other devices and with everyone needing to know.

Figure 3. Left: Wireless Weight Sensor for Semi-Trailers Communicates with Smart Phones. Center: Sound Monitor Wirelessly Alerts Excessive Noise Levels in Work and Public Areas. Right: Digital Controller for Environmental Chamber Allows Smartphone Access. (Intellia, Iotsens, Weisstechnik) [6,7,8]

IoT Connections

Devices like these described so far are one end of the IoT technology stack. They are the public-facing “things,” but just one layer of the stack. The data they send or receive travels up and down layers of connection points and software programs. At the stack’s other end are the device master applications and data storage residing in the datacenter-held cloud.  Because IoT things/devices differ so widely by function, there are many different software platforms in use. The largest telecom and data companies are all active IoT developers in this continuously evolving arena.

Figure 4. The IoT Technology Stack. At Its Base are Billions of Connected Things Whose Data Travels Along Multiple Connections and is Ultimately Managed by Cloud-based Applications. (IoT Business) [9]

Thermal Management Issues in the IoT

Connected consumer IoT devices, or things, are typically very low power. No added thermal management is needed. Power levels are more likely to increase with devices serving the Industrial IoT, though passive cooling, e.g. heat sinks, remedy most heat issues.

Figure 5. Inside an IIoT Air-Cooled Gateway that Provides Long-Range Communications for Monitoring Urban, Farming, and Transportation Systems. (RAKwireless) [10]
Figure 6. A Cold Plate Attached to a Datacenter PCB Directs Jets of Liquid onto Hot Spots on Rack-Stored Processors. (JetCool) [11]

To effectively manage generated heat, cloud-hosting datacenters are using air-cooling, direct-liquid, and immersion cooling (submerged server) systems. As transistor densities increase on smaller package chips, the centers must enhance their thermal management capabilities, while managing power consumption and operation costs.  

Figure 7. Air Cooled Chillers on a Datacenter Rooftop Treat the Water Circulating Through the Center to Cool Its Server Electronics. (Engineered Systems) [12]
Figure 8. Microsoft’s Project Natick Immersed a Datacenter off the Scottish Seashore as a Lower Cost Method for Cooling Its Servers (Microsoft) [13]

IoT – What Comes Next?

These days the IoT is focused on two more letters: AI. In fact, the Artificial Intelligence of Things (AIoT) takes the I (Internet) for granted and enables smart devices and systems to analyze data, make decisions, and act on that data without any interference of humans.

Most AIoT applications are currently retail product-oriented and focused on the implementation of cognitive computing in consumer appliances. For instance, computer vision systems can leverage facial recognition to recognize customers, and compile demographic and preferences data about them. Also, Tesla’s autopilot systems are using radars, sonars, GPS, and cameras to glean data about driving conditions. Then an AI system makes decisions about the data the internet of things devices are collecting to optimize the car’s piloting. [15]

References

  1. DanielElizalde, https://danielelizalde.com/what-is-the-internet-of-things/
  2. Kangaroo, https://heykangaroo.com/products/doorbell-camera-chime
  3. GoodWorkLabs, https://www.goodworklabs.com/apple-watch-app-is-the-smart-watch-revolution-finally-making-sense/
  4. StreamLabs, https://streamlabswater.com/products/streamlabs-water-monitor
  5. PlanetDave, https://planetdave.com/2023/11/a-delightful-drive-the-2023-toyota-crown-platinum/
  6. Biz4intellia, https://www.biz4intellia.com/iot-sensors/
  7. IoTsens, https://www.iotsens.com/en/product/sound-monitor/
  8. Weisstechnik, https://weiss-na.com/product/webseason-the-controller-designed-by-and-for-end-users/
  9. https://iotbusinessnews.com/2022/07/13/86750-what-is-the-iot-technology-stack/
  10. RAKwireless, https://news.rakwireless.com/wisgate-connect-why-did-we-did-it/
  11. JetCool, https://www.datacenterknowledge.com/power-and-cooling/liquid-cooling-adoption-data-centers-becoming-zero-sum-game
  12. Engineered Systems, https://www.esmagazine.com/articles/100400-air-cooled-chillers-are-back-in-data-centers-and-they-mean-business
  13. Microsoft, https://news.microsoft.com/source/features/sustainability/project-natick-underwater-datacenter/
  14. Tealcom, https://tealcom.io/post/the-intersection-of-ai-iot-and-connectivity/
  15. IndustryWired, https://industrywired.com/how-artificial-intelligence-can-help-manage-flood-of-iot-data/

Leave a comment

Posted in Artificial Intelligence, Automotive, thermal management

Tagged , , , , , ,

Thermal Management for AI Chips

Posted on February 20, 2025 by | 2 comments

The use of artificial intelligence (AI) programs is growing very quickly, despite some concerns and precautions. It’s being spurred by powerful new hardware from companies like Nvidia, and by new lower-cost, open source large language model (LLM) software like those from DeepSeek.

Likewise, AI chip sales are soaring, and more powerful and specialized AI chips are being steadily introduced. At this writing, Nvidia, the leading AI chip provider is now the third-most-valuable company in the world, valued at over $2.2 trillion. Nvidia is both developing new AI chips and acquiring smaller AI companies that design processors and develop AI applications. [1]

Figure 1. The Nvidia A100 AI Chip and DeepSeek’s Free to Low-Cost AI Model are Major Reasons for AI’s Fast-Growing Deployment and Use in Nearly All Industries. (Nvidia, DeepSeek) [2]

Other chip companies are also thriving. Micron Technology is reporting record sales, much of them from supplying memory chips to Nvidia. AMD provides chips that rival Nvidia’s flagship AI machine learning chip. And Intel is getting $8.5 billion from a US federal program (CHIPS) to support its goal to build the largest AI chip manufacturing site in the world. [3,4]

Innovative Chips Bring High Heat

AI chip technology, which evolved from the graphics processing units, GPUs, developed for the data needs of video games, may be the most understood part of the AI world. But this evolution is remarkable. The Nvidia A100 AI Chip features 54 billion transistors. By comparison, an AMD Ryzen 7 1700 gaming processor for a contemporary PC has 4.8 million transistors. [5]

By leveraging parallel processing capabilities, AI chips effectively handle large datasets, allowing multiple tasks to be executed simultaneously. These chips interact with special ASICs, FPGAs, TPUs and VPUs to perform machine learning and neural network processing. The AI networks can solve complex algorithms and are teaching computers to process data in a way that is inspired by the human brain.

For example, AI can use inference – combining reasoning and decision-making based on available information – to apply real-world knowledge for facial recognition, gesture identification, natural language interpretation, image searching  and much more. [6]

Figure 2. Intel FPGAs Support Real-Time Deep Learning Inference for Embedded Systems and Data Centers. (Intel/Mirabilis) [7,8]

AI chips demand high power to support increased processing demand. As a result, excessive waste heat can degrade performance or trigger system failure. AI system designers depend on thermal management solutions to manage AI processor temperatures. Cooling resources at both chip-level and facility, e.g. data center scale are needed to keep AI chips functioning at proper temperatures.

Liquid Cooling AI Chips

The heavy lifting in AI processing is done in data centers, which are the focus of most technical developments. Their high concentration of high-power chips presents formidable heat management challenges, especially when the thermal design power of the GPU has increased over the past two decades, rising from 150 watts to more than 700 watts.

Now consider the recently unveiled 1200 watt Nvidia Blackwell B200 tensor core chip—the company’s most powerful single-chip GPU, with 208 billion transistors—which Nvidia says can reduce AI inference operating costs (such as running ChatGPT) and energy. Two of these B200 chips are combined with an Nvidia Grace CPU to complete the newly released, even higher-performing GB200. Its total projected power draw: up to 2,700 watts.

Figure 3. The Nvidia GB200 NVL72 Data Center Computer System Combines 36 GB200s (72 B200 GPUs and 36 Grace CPUs total). (Nvidia) [9]

The GB200 chip is a key part of Nvidia’s new GB200 NVL72, a liquid-cooled data center computer system designed specifically for AI training and inference tasks. Amazon Web Services, Dell Technologies, Google, Meta, Microsoft, OpenAI, Oracle, Tesla, and xAI, are expected to adopt the Blackwell platform. [9]

The ever increasing number of transistors attached to data center PCBs translates to higher performance but also more heat than ever before. Liquid cooling systems, like in the Nvidia data center system can significantly reduce energy consumption. This leads to lower operating expenses in the long run. It also produces less noise and, for direct on chip cooling, takes up less space. 

Direct to chip or node cooling involves circulating a coolant directly over heat-generating components, including AI chips. This method significantly increases cooling efficiency by removing heat directly at the source. These systems can use a variety of coolants, including water, dielectric fluids, or refrigerants, depending on the application’s needs and the desired cooling capacity. [9]

Figure 4. Direct-to-Chip Liquid Cooling. Note the Small Heat Sinks to Air-Cool Other Components. (AnD Cable Products) [10]
Figure 5. Data Center Racks Cooled by Lenovo’s Direct to Node Liquid Cooling System. (ServeTheHome) [11]

Immersion cooling takes liquid cooling a step further by submerging the entire server, or parts of it, in a non-conductive liquid. This technique can be highly efficient as it ensures even and thorough heat absorption from all components. Immersion cooling is particularly beneficial for high-performance computing (HPC) and can dramatically reduce the space and energy required for cooling.

Figure 6. Immersion Cooling Submerges All or Parts of a Server in a Non-Conductive Liquid Coolant. (GIGABYTE) [12]

Air Cooling AI Chips

Nvidia’s Jetson chips bring accelerated AI performance to IoT and Edge applications in a power-efficient and compact form factor (smaller than 100mm x 100mm). Less power-consuming (up to 75 watts) than data center AI chips, thermal management is still needed. Jeston components are typically cooled with heat sinks, which can be configured as active (with attached fan) or passive (fanless). 

Figure 7. Passive (Fanless) and Active (Fan-assisted) Heat Sinks Designed to Cool Nvidia Jeston AI Chips in Embedded and Edge Devices.
(Advanced Thermal Solutions, Inc.) [13]
Figure 8. The Edge Ultrastar Transportable Edge Server from Western Digital Features a 70 Watt Nvidia Tesla GPU Cooled by Four Internal 60mm Fans. (Western Digital) [14]

Conclusion

Artificial intelligence is hot in the marketplace. So are AI chips. As complex as they are, simply surpassing a heat threshold can affect their proper function. Their thermal management is essential.

Figure 9. AI Deployment in Industry will Generate Strong Demand for Smart Devices.
One Example is Sensors of All Types. (Fierce Electronics) [15]

The world will experience the impacts of artificial intelligence. Like the Internet and mobile technology, it will become pervasive, far beyond deep fakes and term papers, instead driving development of more capable tools for industry, medicine and more, and for managing our daily live3. With care, this revolution should be benign, and greatly improve our lives and our world.
 
References
1. Motley Fool, https://www.fool.com/investing/2024/03/21/nvidia-just-bought-5-ai-stocks-2-stand-out-most/
2. OPB, https://www.opb.org/article/2025/02/05/international-regulators-probe-how-deepseek-is-using-data-is-the-app-safe-to-use/
3. Yahoo Finance, https://finance.yahoo.com/news/amd-dethrone-nvidia-artificial-intelligence-112400772.html
4. Quartz, https://qz.com/intel-ai-chip-factory-world-chips-act-funds-1851358125
5. Nvidia, https://www.nvidia.com/en-us/data-center/a100/
6. OurCrowd, https://www.ourcrowd.com/learn/what-is-an-ai-chip
7. Intel, https://www.intel.com/content/www/us/en/software/programmable/fpga-ai-suite/overview.html
8. Mirabilis Design, https://www.mirabilisdesign.com/intel-fpga-neural-processor-ai/
9. Ars Technica, https://arstechnica.com/information-technology/2024/03/nvidia-unveils-blackwell-b200-the-worlds-most-powerful-chip-designed-for-ai 
10. AnD Cable Products, https://andcable.com/data-center-trends/data-center-liquid-cooling/
11. ServeTheHome, https://www.servethehome.com/lenovo-sd650-v2-and-sd650-n-v2-liquid-cooling-intel-xeon-nvidia-a100-neptune/
12. GIGABYTE, https://www.gigabyte.com/Solutions/gigabyte-single-phase
13. Advanced Thermal Solutions, Inc., https://www.qats.com/eShop.aspx?q=Device%20Specific%20-%20NVIDIA
14. Western Digital, https://www.westerndigital.com/en-ap/products/data-center-platforms/ultrastar-transporter?sku=1ES2562
15. Fierce Electronics, https://www.fierceelectronics.com/components/sensors-artificial-intelligence-and-concepts-you-may-want-to-know-i

2 Comments

Posted in Artificial Intelligence, Liquid Cooling, thermal management

Tagged , , ,