Immersion Liquid Cooling of Servers in Data Centers

A data center is a large infrastructure used to house large quantities of electronic equipment, such as computer servers, telecommunications equipment, and data storage systems, etc. The data center requires non-interrupted power, communication and internet access to all equipment inside, it also has dedicated environment control system which provides appropriate working conditions for the electrical devices hosted inside.

Immersion Cooling

Traditional data centers use cold air generated by a room air conditioner system (CRAC) to cool the servers installed on the racks. Cooling the electrical devices by cold air generated by an air conditioner is an easy method to implement. However, it is not a very efficient method in terms of power consumption.

The inefficiency of the method can be contributed to several causes: generating and delivering cold air from a chiller to servers is a multiple heat transfer process, such as the mixing of warm and cool air in the room, which reduces the efficiency and power consumption of cooling hardware such as chillers, computer room air conditioners (CRACs), fans, blowers and pumps.

Data center designers and operators have invented many ways to improve the data center’s thermal efficiency, such as optimizing the rack layout and air conditioner location, separating cold aisles and hot aisles, optimizing the configuration of pipes and cables in under-floor plenum, introducing liquid cooling to high-power severs.

While the above methods can improve the data center heat load management, they cannot dramatically reduce the Power Usage Effectiveness (PUE), which is a measure of how efficiently a datacenter uses its power and is defined as the ratio of total datacenter power consumption to the IT equipment power consumption.

An ideal PUE is 1,0. A better way, proposed and used by some new data centers, is directly bringing the outside cold air to the servers. This method can eliminate the computer room air conditioners (CRACs). To achieve this, the data center has to be located in a specific area where cold air can be provided for all four seasons and the servers have to have higher operating environmental temperature.

Another dramatic solution proposed and used by some companies is liquid immersion cooling for entire servers. When compared with traditional liquid cooling techniques, the liquid immersion cooling uses dielectric fluid as a working agent and open bath design. This eliminates the need for hermetic connectors, pressure vessels, seals and clamshells. There are several different liquid immersion cooling methods.

This article will review the active single-phase immersion cooling technology proposed by Green Revolution Cooling (GRC) [1] and a passive two-phase immersion cooling technology proposed by the 3M Company [2].

Green Revolution Cooling has designed a liquid-filled rack to accommodate the traditional servers and developed dielectric mineral oil as the coolant. Figure 1 shows the liquid cooling racks with chiller and an inside view of a CarnotJet cooling rack from GRC. The racks are filled with 250 gallons of dielectric fluid, called GreenDEF™, which is a non-toxic, clear mineral oil with light viscosity.

Figure 1. Server racks and chiller (left) and inside view of the server rack. [1]

The servers are installed vertically into slots inside the rack and fully submerged in the liquid coolant. Pumps are used to circulate the cold coolant from the chiller to the rack. The coolant returns to the chiller, after removing heat from the servers. Because of its high heat capacity and thermal conductivity, the GreenDEF™ can cool the servers more efficiently than air.

The server racks are semi-open to the environment and the coolant level is constantly monitored by the system. Figure 2 shows a server motherboard is being submerged in the coolant liquid inside a server rack from GRC.

Figure 2. A Server Motherboard Being Immersed in Liquid Coolant in A Server Rack. [1]

Intel has conducted a year-long test with immersion cooling equipment from Green Revolution Cooling in New Mexico [3]. They have found that the technology is highly efficient and safe for servers. In their tests, Intel tested two racks of identical servers – one using traditional air cooling and the other immersed in a Green Revolution enclosure. Over the course of a year, the submerged servers had a partial Power Usage Effectiveness (PUE) of 1.02 to 1.03, equaling some of the lowest efficiency ratings reported using that metric.

The 3M Company is also actively engaged in immersion cooling technology and has developed a passive two-phase immersion cooling system for servers. Figure 3 illustrates the concept of the immersion cooling system developed by 3M. In a specially designed server rack, servers are inserted vertically in the rack. The servers are immersed in 3M’s Novec engineered fluid, a non-conductive chemical with a low boiling point.

The elevated temperature of electronic components on the sever boards will cause the Novec engineered fluid to boil. The evaporation of the fluid will remove a large amount of heat from the heated components with small temperature difference. The evaporated fluid travels to the upper portion of the server rack, where it condenses to liquid on the surface of the heat exchanger cooled by the cold water. The condensed liquid flows back to the rack bath, driven by the force of gravity. In 3M’s server rack, the liquid bath is also semi-open to the outside environment.

Because the cooling method is passive, there is no pump needed in the system.

Figure 3. Passive Two-phase Immersion Cooling System from 3M. [2]

By utilizing the large latent heat of Novec engineered fluid during evaporation and condensation, the coolant can remove heat from servers and dissipate it to water heat exchanger with small a temperature gradient. To enhance the boiling on the component surfaces, 3M invented special coating for electronic chips inside the liquid bath. The boiling enhancement coating (BEC) is a 100 mm thick porous metallic material.

The application of the BEC is illustrated in Figure 4. The coating is directly applied to the integrated heat spreader (IHS) of the chip. Tuma [2] claimed that the coating can produce boiling heat transfer coefficients in excess of 100,000 W/m2-K, at heat fluxes exceeding 300,000 W/m2.

Figure 4. Application of boiling enhanced coating (BEC). [2]

In his paper, Tuma [2] discussed the economic and environmental merits of the passive two-phase immersion cooling technology for cooling data center equipment. He concluded that liquid immersion cooling can dramatically decrease the power consumption for cooling relative to traditional air-cooling methods. It can also simplify facility construction by reducing floor space requirements, eliminating the need for air cooling infrastructure such as plenum, air economizers, elevated ceilings etc.

Green Revolution Cooling and 3M have demonstrated the feasibility and applicability of using immersion cooling technology to cool the servers in data centers. The main advantages of immersion liquid cooling are saving overall cooling energy and maintaining the component temperature low and uniform. However, both immersion liquid cooling technologies require specially designed sever racks. Specially formulated coolants are needed for both cooling technologies, too, and they are not cheap. For the traditional air-cooled data center, the air is free, abundant and easy to deliver.

In both immersion cooling technologies, the servers have to be vertically installed inside the server rack, which will reduce the date center footprint usage efficiency. Because the liquid baths used in immersion cooling are open to the environment, coolant is gradually and inevitably lost to the ambient during long term service.

The environmental impact of the discharge of a large amount of coolant by data centers has to be evaluated, too. The effect of the coolant on the connectors and materials used on the PCB is not also very clear.

Immersion liquid cooling is a very promising technology for cooling high-power servers. But, there are still obstacles that need to be overcome before their large scale application is assured.   

References

  1. http://www.grcooling.com
  2. Tuma, E. P., “The Merits of Open Bath Immersion Cooling of Datacom Equipment,” 26th IEEE SEMI-THERM Symposium, Santa Clara, California, USA  2010.
  3. http://www.datacenterknowledge.com

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.

One response to “Immersion Liquid Cooling of Servers in Data Centers

  1. The information in this article is very outdated. GRC’s coolant was rebranded to ElectroSafe back in 2015, it is no longer mineral oil based but rather a synthetic base stock. Also, single-phase coolants, as the name suggests, do not evaporate, so you don’t lose coolant to the environment. GRC already has large scale deployments across the globe (13 countries) and one of it’s longest-running customers, CGG in Houston has been running efficiently for close to 9 years with the same racks, CDUs, and coolant, which has seen multiple generations of hardware.
    I agree that air is free and abundant, but it’s also a bad conductor of heat. Also, the refrigirants used in chillers aren’t free, neither is the energy required to chill and move the amount of air needed to cool data centers.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.