Author Archives: Josh Perry

Industry Developments: Extrusion Profile Heat Sinks

By Norman Quesnel
Senior Member of Marketing Staff
Advanced Thermal Solutions, Inc. (ATS)

Extruded metal heat sinks are among the lowest cost, widest used heat spreaders in electronics thermal management. Besides their affordability, extruded heat sinks are lightweight, readily cut to size and shape, and capable of high levels of cooling.

Metal Choices

Most extruded heat sinks are made from aluminum alloys, mainly from the 6000 alloy series, where aluminum dominates. Trace amounts of other elements are added, including magnesium and silicon. These alloys are easy to extrude and machine, are weldable, and can be hardened.

Common alloys for extruded heat sinks are the 6063 metals. These can be extruded as complex shapes, with very smooth surfaces. 6061 aluminum is also used for extrusions. Its tensile strength (up to 240 MPa) is superior to 6063 alloys (up to 186 MPa). In addition to heat sinks, these aluminum alloys are popular for architectural applications such as window and door frames. [1]

Extrusion Profile Heat Sinks
Figure 1. An extruded aluminum heat sink with a black anodized finish. [2]

The surfaces of these metals can be anodized to replace their naturally occurring surface layer of aluminum oxide. Anodizing provides more heat transfer, corrosion resistance and better adhesion for paint primers. It is an electrochemical process that increases surface emissivity, corrosion and wear resistance, and electrical isolation.

The Extruding Process

Aluminum alloys are popular for extruding as heat sinks because they provide both malleability and formability. They can be easily machined and are as little as one-third the density of steel. This results in extrusions that are both strong and stable, at a reduced cost relative to other materials.

Figure 2. Heated aluminum alloy billets are pushed through a die to produce extruded length heat sinks and other parts. [3]

The aluminum extrusion process starts with designing and creating the die that will shape the heat sink profile. Once this has been done, a cylindrical billet of aluminum is heated up in a forge to high temperatures, generally between 800-925°F (427-496°C). Next, a lubricant is added to the aluminum to prevent it from sticking to any of the machinery. It is then placed on a loader and pressure is applied with a ram to push heated aluminum through the die.

During this process, nitrogen is added in order to prevent oxidation. The extruded part will pass completely through the die and out the other side. It has now been elongated in the shape of the die opening. The finished extrusion is then cooled, and if necessary, a process of straightening and hardening creates the finished product.

They can be cut to the desired lengths, drilled and machined, and undergo a final aging process before being ready for market. [4]

Finished heat sinks typically come with anodized surfaces, which can enhance their thermal performance. Alternatively, a chromate finish provides some corrosion protection, or can be used as a primer before a final paint or powder coating is applied. [5]

Shapes of Extruded Heat Sinks

Extruded heat sink profiles range from simple flat back fin structures to complex geometries for optimized cooling. They can be used for natural (passive) or forced convection (active) with an added fan or blower. Extruded profiles can also include special geometries and groove patterns for use with clip or push pin attachment systems. [6]

Figure 3. Extruded heat sinks are available in many shapes and lengths. [6]

Extrusions are also available in bulk lengths, e.g. 8 feet, which can be cut to different lengths per customer needs.

Optimizing Thermal Performance

6063 aluminum alloy has a thermal conductivity of 201-218 W/(mK). Higher tensile strength 6061 aluminum’s thermal conductivity ranges from 151-202 W/(mK).

Besides choosing the aluminum alloy, selecting an optimal extruded heat sink should factor in its overall dimensions and weight, the specified thermal resistance, and the extrusion shape (flat-back, flat-back with gap, hollow, double-sided, etc.). [7]

Extruded heat sinks can be designed with very thin, and thus more, fins than other sink types. They can be extruded with aspect ratios of around 8:1, which can greatly optimize heat sink performance. A heat sink’s aspect ratio is basically the comparison of its fin height to the distance between its fins.

In typical heat sinks the aspect ratio is between 3:1 and 5:1. A high aspect ratio heat sink has taller fins with a smaller distance between them for a ratio that can be 8:1 to 16:1 or greater.

Figure 4. Different thin fin extruded heat sinks mounted on a PCB. [8]

Thus, a high aspect ratio heat sink provides greater density of fins in a given footprint than with a more common size sink. The great benefit is the increased amount of heat dissipating surface area it provides due to its additional fins. Further, these heat sinks do not occupy any more length or width. The result is a more efficient heat sink with higher performance per gram in the same footprint. [9]

Applications

An extruded heat sink is used mainly to increase the surface area available for heat transfer from high-power semiconductor devices, thus reducing a given semiconductor’s external case temperature, as well as its internal junction temperature.

Figure 5. Extruded heat sinks mounted on processors by clips (left) and push pins (right). [10]

This allows the semiconductor devices to perform at their highest level, with maximum reliability. Such semiconductor devices include (but are not limited to) RF power transistors, RF power amplifiers, Power MOSFETs, IGBTs, inverter power modules, and thyristor modules.

Figure 5. Extruded heat sinks screwed onto a brick DC-DC converter. [11]

In some power conversion circuit applications, large diodes, rectifiers, diode modules and even high-power resistors (thick film, etc.) can also require thermal contact with an extruded heat sink. For cooling DC-DC power converters and power modules, extruded heat sinks are available for full, half, quarter and one-eighth brick sizes

References

  1. https://en.wikipedia.org/wiki/6063_aluminium_alloy
  2. http://www.aluminumextrusionsprofiles.com/sale-7552970-black-anodized-aluminium-heat-sink-profiles-extruded-aluminum-heatsink-radiators.html
  3. https://www.aec.org/page/basics_basics
  4. https://www.clintonaluminum.com/6061-aluminum-vs-6063-in-extrusion-applications
  5. http://www.wakefield-vette.com/products/natural-convection/thermal-extrusions-overview/CategoryID/15/Default.aspx
  6. https://www.boydcorp.com/thermal/heat-sinks/extruded.html
  7. http://www.richardsonrfpd.com/Pages/Product-End-Category.aspx?productCategory=10188
  8. http://www.getecna.com/products/heat-sinks/
  9. https://www.qats.com/cms/2013/04/11/increased-performance-from-high-aspect-ratio-heat-sinks/
  10. https://www.qats.com/Heat-Sink/Attachments
  11. https://www.powerelectronics.com/thermal-management/heat-sinks-cool-brick-dc-dc-converters

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com. To register for Qpedia and to get access to its archives, visit 
https://www.qats.com/Qpedia-Thermal-eMagazine.

Application of Air Jet Impingement for Cooling a 1U System

As their speeds increase, the heat dissipation from high performance processors requires more innovative cooling techniques for heat removal. One such technique, jet impingement, provides one of the highest heat transfer coefficients among cooling methods. This property of jet impingement has been used by Advanced Thermal Solutions, Inc. (ATS) in a 1-U server application. Jet impingement has also been applied to ATCA chassis.

Air Jet Impingement
Cooling high-powered 1U servers requires new thermal management techniques, including advanced air jet impingement. (Wikimedia Commons)

A Ujet-1000™ 1-U chassis made by ATS was assessed in the company’s thermal test lab. Four identical heat sinks were tested under conventional parallel flow and ATS’ proprietary Therm-Jettimpingement flow. The results showed a 20-40 percent improvement in the thermal performance of the heat sinks.

The Ujet-1000™ is a 1 to 2 KW 1-U chassis system (depending on component case and ambient temperature) designed for the most demanding, telecom and server applications. Lab tests demonstrated that four heat sinks on four simulated chips located on a PCB achieved 0.16 to 0.18oC/W thermal resistance. The power dissipation of each simulated component was maintained at 200 W. On the other hand, the thermal resistance of the same heat sinks with parallel flow, using the same fans, was almost 20 to 40% worse.

The new ATS Therm-Jett technology uses a specially made duct with an impingement plate beneath it to create jet impingement on top of the components and heat sinks. The tremendous increase in heat transfer coefficient leads to significant reduction of thermal resistance compared to the other conventional 1-U systems. A Therm-Jett system can be built for any specific configuration. The impingement duct is less than 5 mm thick and is located in the chassis on top of the motherboard. In addition to high heat transfer coefficient, fresh air is distributed between all heat sinks at inlet temperature.

In contrast, in conventional cooling systems, the upstream heat sinks and components receive air at inlet temperatures which are cold and gradually warm up as the air moves downstream. The increase of air temperature effectively reduces the cooling effect of the air downstream.

The other advantage of Therm-Jett™ is that there is no need to make a special duct for each heat sink, thus freeing the motherboard for other components. Even by adding ducts, other components such as memory cards, resistors and capacitors located upstream of the heat sinks on the PWB would deprive the heat sink of the flow at its most critical point, which is close to the base.

Figure 1 shows a CAD drawing of the real system under test. The cooling is provided by eight 40 mm high capacity double fans located in the midsection of the 1-U chassis. The power to the heat sinks was provided by four heaters attached below and dissipating 200 W each. The heat dissipation of the power supply was simulated by attaching a rectangular heater strip under the power supply which dissipates about 100 W. A “U” shape frame made of aluminum was located under the hard drives. The power of four hard drives was simulated by placing a rectangular heater under the “U” shape frame which dissipates about 80 W.

Four thermocouples were placed in holes at the center of the base of the heat sink downstream. The holes were filled with thermal grease to minimize the interfacial resistance. Three thermocouples were attached to the aluminum “U” frame, and their average temperature was recorded as an approximate temperature of a real hard drive. One thermocouple was also attached to the base of the power supply to measure its approximate temperature. All temperature measurements were taken using J type thermocouples.

Figure 1. Schematic of the Ujet-1000TM and the Therm-JETT™ cooling duct. (Advanced Thermal Solutions, Inc.)

Figure 2 shows the exploded view of the Ujet-1000™ chassis.

Figure 2. Exploded view of the Ujet-1000TM and the Therm-JETT™ cooling duct. (ATS)

Figure 3 shows a conventional cooling system for a 1-U system. In this system, the air flow from the fans is parallel to the heat sinks.

Figure 3. Conventional Cooling System in a 1-U Application. (ATS)

Results

Figure 4 shows the schematic set up of the conventional cooling system. In this configuration, eight (8) blowers move the air in parallel to the heat sink fins.

Figure 4. Configuration of conventional layout in a 1U system for temperature measurement. (ATS)

Figure 5 shows the implementation of an ATS Therm-Jet, which provides jet impingement on the same four heat sinks, and the location of impingement holes with respect to the heat sinks.

Figure 5. Configuration of an ATS Therm-JETT™ application in a 1U system for temperature measurement. (ATS)

Tables 1 and 2 show the experimental values obtained within a 1-U chassis made by ATS. The two sets of tests were done for both 12 and 6 volts to the fans. The thermal resistance data of all four heat sinks, hard drives and the power supply were obtained for both conventional cooling and jet impingement cases. The acoustic noise for each case was also recorded for comparison.

The data shows an improvement of thermal resistance of 22% to 42% for the heat sinks from jet impingement as compared to conventional cooling. The power supply shows a 10% improvement in the thermal resistance.

The hard drive, though, shows a 20% degradation. This is due to the fact that, with jet impingement, the pressure drop on the fans increases, consequently decreasing the flow through the system. However in an actual system the percentage will be smaller. That’s because the heat generated will be more volumetric compared to the current setup where heat is generated on the surface of the “U” shape aluminum piece located at the bottom of hard drives.

In that case, the decrease of flow through the system will have less impact. Additionally, the increase in hard drive temperature is less than 2°C in this experiment, which is generally not large enough to be a concern.

Table 1. Experimental test results comparing conventional and Therm-JETT™ results with 12 volts to the fans.
Table 2. Experimental test results comparing conventional and Therm-JETT™ results with six volts to the fans.

The question might be raised as to whether the performance of the heat sinks could be improved if we removed the impingement duct, increased the heat sink height by the height of impingement duct and ducted the flow.

We analyzed this situation and found that the improvement would be at most 5%, if we assume that the heat sink is ducted and the pressure drop is the same in both short and tall versions. To study this problem in detail one must consider the fan curves instead of using a fixed volumetric flow rate. Interested readers will find an article in a previous issue of the ATS Qpedia Thermal eMagazine [1] with more information about this topic.

Table 3 shows the temperatures of the four heat sinks, the hard drive and the power supply. As we mentioned earlier, the heat sinks were mounted on 200 W devices, the power supply was dissipating 100 W and the hard drives were dissipating 80 W. The results are shown for jet impingement and conventional parallel air flow over 23.5 mm tall heat sinks, and ducted flow over heat sinks with 28.5 mm tall heat sinks. It can be seen that heat sink temperatures are significantly lower for jet impingement even compared with a taller heat sink with ducted flow.

Table 3. Comparison of temperatures for jet impingement and parallel flow with 23.5 mm heat sinks and ducted flow with 28.5 mm tall heat sinks.

Table 4 shows the improvement in temperature of the four processors between the jet impingement and the two cases of 23.5 mm heat sink and the ducted 28.5 mm heat sinks. By comparing the results for 6 and 12 volts to the fans, it can be seen that at lower voltage the jet impingement temperature difference is even more than with higher voltage to the fans. This implies low pressure drop fans can significantly benefit from the application of jet impingement.

Table 4. Comparison of temperature improvement for jet impingement and parallel flow with 23.5 mm heat sinks and ducted flow for 28.5 mm tall heat sinks.

Figure 6 is a graphical representation of Table 4. The figure shows the significant temperature increases in the case of parallel flow and ducted flow for the heat sinks compared to jet impingement technology. Components (heat sinks) 2 and 3 are hotter than components 1 and 2 because they are downstream and the approach air temperature is higher for a ducted flow. In the jet impingement mode, the impingement flow is at upstream temperature and therefore much cooler than the air received in ducted flow.

In impingement mode, there is another flow coming axially toward the components, called cross flow. It is the interaction of cross flow and impingement that causes the cooling of the component (heat sink).

Figure 6. Heat sink temperature increase of parallel and ducted flow compared to jet impingement cooling. (ATS)

It should be noted that the above experiment was done for a heat source that is the same size as the heat sink base; hence, the spreading resistance is zero because it is almost independent of the heat transfer coefficient. The spreading resistance can be added to the above numbers for other sizes of heat sources.

The same concept of jet impingement has been applied to simulated components in ATCA chassis. The results will be published in subsequent Qpedia articles. The data improvement is promising. Even though conventional air-cooling technology is fast approaching its thermodynamic limit, there are still numerous potentials for air cooling which will enable this technology to be used in the years to come.

Reference

1.  “Heat Sink Thermal Resistance as a Function of Height-Ducted Flow with Fan Curve,” Qpedia Thermal eMagazine, Advanced Thermal Solutions, Inc., January 2009.


For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.

April Webinar on Thermal Interface Materials

Advanced Thermal Solutions, Inc. (ATS) is hosting a series of monthly, online webinars covering different aspects of the thermal management of electronics. This month’s webinar will be held on Thursday, April 25 from 2-3 p.m. ET and will cover the use of thermal interface materials to enhance heat sink performance. Learn more and register at https://qats.com/Training/Webinars.

Meeting the thermal management requirements of high-performance servers

High-performance servers are devices specially designed to handle large computational loads, a huge amount of communication signals, fast data processing, etc. Due to their task-oriented nature, high-performance servers must have high reliability, interchangeability, compact size and good serviceability.

High-Performance Servers

To achieve high computational speed, high-performance servers generally have dozens of CPUs and memory models. They also have dedicated data process modules and control units to ensure seamless communication between CPUs and parallel data processing ability. To reach higher speeds, the power dissipation of high–performance CPUs has been increasing continuously in the past decade for its use in high-performance servers.

Cooling dozens of kW servers brings a unique challenge for thermal engineers. To deal with the ever-growing high heat flux issue in high-performance servers, it will need the cooperation of electrical, mechanical and system engineers to solve the problem. The job to remove the high heat flux from CPUs to ambient requires chip level, board level and cabinet level solutions.

Wei [1] described Fujitsu’s thermal management advancements in their high-end UNIX server PRIMEPOWER 2500. The server cabinet is shown in Figure 1. Its dimension is 180cm × 107cm × 179cm (H×W×D) and has a maximum power dissipation of 40 kW. The system configuration of PRIMEPOWER 2500 is shown in Figures 2 and 3. It has 16 system boards and 2 input/output (I/O) boards installed vertically on two back-panel boards. The two back-panel boards are interconnected by six (6) crossbars installed horizontally.

Figure 1. PRIMEPOWER 2500 Cabinet [1]
Figure 2. PRIMEPOWER 2500 System Configuration [1]
Figure 3. PRIMEPOWER 2500 System Board Unit [1]

To cool the electrical components inside PRIMEPOWER 2500, 48 200-mm-diameter fans are installed between the system board unit and the power supply unit. They provide forced air cooling for system boards and power supplies. In addition, six 140-mm-diameter fans are installed on one side of crossbar to cool the crossbar boards with a horizontal flow. The flow direction is shown in Figure 3. Each system board is 58 cm wide and 47 cm long.

There are eight CPU processors, 32 Dual In-Line Memory Modules, 15 system controller processors, and associated DC-DC converters on each system board. The combined power dissipation per system board is 1.6 kW at most.

Figure 4. PRIMEPOWER 2500 System Board [1]

To cool the electrical components inside PRIMEPOWER 2500, 48 200-mm-diameter fans are installed between the system board unit and the power supply unit. They provide forced air cooling for system boards and power supplies. In addition, six 140-mm-diameter fans are installed on one side of crossbar to cool the crossbar boards with a horizontal flow. The flow direction is shown in Figure 3. Each system board is 58 cm wide and 47 cm long.

There are eight CPU processors, 32 Dual In-Line Memory Modules, 15 system controller processors, and associated DC-DC converters on each system board. The combined power dissipation per system board is 1.6 kW at most.

Forced air-cooling technology is commonly used in computers, communication cabinets, and embedded systems, due to its simplicity, low cost and easy implementation. For high-performance servers, the increasing power density and constraints of air-cooling capability and air delivery capacity have pushed forced air cooling to its performance limit.

For high power systems like PRIMEPOWER 2500, it needs a combination of good CPU design, optimized board layout, advanced thermal interface material (TIM), high-performance heat sinks, and strong fans to achieve desired cooling.

The general approach to cool the multi-board system is first to identify the hottest power component with the lowest temperature margin. For the high-performance server, it is the CPUs. For multiple CPUs on a system board, generally, the CPU located on downstream of a board or other CPUs has the highest temperature.

So, the thermal resistance requirement for this CPU is:

Where Tj,max is the allowed maximum junction temperature, Ta is the ambient temperature, ∆Ta is the air temperature rise due to preheating before the CPU, and qmax is the maximum CPU power.

The junction-to-air thermal resistance of the CPU is:

Where Rjc is the CPU junction-to-case thermal resistance, RTIM is the thermal resistance of thermal interface materials, and Rhs is the heat sink thermal resistance. To reduce the CPU junction temperature, it is critical to find intuitive ways to minimize Rjc, RTIM, and Rhs, because any reduction in thermal resistance is important in junction temperature reduction.

The CPU package and heat sink module of PRIMEPOWER 2500 are shown in Figure 5. The CPU package has an integrated heat spreader (IHS) attached to the CPU chip. A high-performance TIM is used to bond the CPU chip and IHS together, see Figure 6. The heat sink module is mounted on the IHS with another TIM in between.

Figure 5. PRIMEPOWER 2500 CPU Package and Heat Sink Module [1]
Figure 6. CPU Package [1]

The TIM used in between the CPU chip and the IHS are crucial to the CPU’s operation. It has two key functions: to conduct heat from the chip to the IHS and to reduce the CPU chip stress caused by the mismatch of the coefficient of thermal expansion (CTE) between the CPU chip and IHS. Fujitsu developed a TIM made of In-Ag composite solder for the above application. The In-Ag composite has a low melting point and a high thermal conductivity. It is relatively soft, which is good for absorbing thermal stress between the chip and the IHS.

Wei [2] also investigated the impact of thermal conductivity on heat spread performance. He found a diamond composite IHS (k=600 W/(mK)) would result in a lower temperature gradient across the chip and low temperature hot spots, compared with aluminum nitride (k=200 W/(mK)) and copper (k=400 W/(mK)). The simulation results are shown in Figure 7.

Figure 7. Heat Spreader Material Comparison [2]

In high-performance servers like the PRIMEPOWER 2500, the thermal performance gains by optimizing the TIM and the IHS are small, because they compose only a small portion of the total thermal resistance. Heat sinks dissipate heat from the CPU to air and have an important role in the thermal management of the server. In a server application, the heat sink needs to meet not only the mechanical and thermal requirements, but also the weight and volume restraints. Hence, heat pipes, vapor chambers, and composite materials are widely used in place of high-performance heat sinks.

Koide et al [1] compared the thermal performance and weight of different heat sinks for server application. The results are shown Figure 8. They used the Cu-base/AL-fin heat sink as benchmark. Compared with the Cu-base/AL-fin heat sink, the Cu-base/Cu-fin heat sink is 50% heavier and gains only 8% performance.

If the heat pipe is used in base, the heat sink weight can be reduced by 15% and the thermal performance increases by 10%. If the vapor chamber is embedded in the heat sink base, it reduces the heat sink weight by 20% and increases the heat sink performance by 20%.

Figure 8. Thermal Performance and Weight Comparison of Different Heat Sinks [1]
Figure 9. (a) USIII Heat Sink for Sun Fire 15K Server, (b) USIV Heat Sink for Sun Fire 25K [3]

Sun Microsystems’ high-performance Sun Fire 15K Server uses USIII heat sink to cool its 72 UltraSparc III (USIII) processors. In Sun Fire 25K Server, the CPUs are upgraded to UltraSparc IV (USIV), which has a maximum power of 108 W. To cool the USIV processor, Xu and Follmer [3] designed a new USIV heat sink with copper base/copper fin, see Figure 9. The old USIII heat sink has 17 forged aluminum fins, the USIV heat sink has 33 copper fins. Both heat sinks have the same base dimensions and height.

Figure 10. Thermal Resistance Comparison between USIII Heat Sink and USIV Heat Sink [3]

Figure 10 shows the thermal resistance comparison between the USIII heat sink and the USIV heat sink. The thermal resistance of the USIV heat sink is almost 0.1°C/W lower than that of the USIII heat sink at medium and high flow rates, which is a huge gain in thermal performance. The thermal performance improvement of the USIV heat sink is not without penalty.

Figure 11. Pressure Drop Comparison between USIII Heat Sink and USIV Heat Sink [3]

Figure 11 shows the pressure drop comparison between the USIII heat sink and the USIV heat sink. For the same air flow rate, the pressure drop of the USIV heat sink is higher than that of the USIII heat sink. That means the Sun Fire 25K Server needs stronger fans and better flow arrangements to ensure the USIV heat sinks have adequate cooling flow.

The design of the cooling method in high-performance servers follows the same methodology used in the design cooling solution of other electronic devices, but at an elevated scale. The main focus is to identify the hottest components, which in most cases is CPUs. Due to extreme high power of CPUs, memory modules, cheat spreader, TIM, and heat sinks to achieve desired cooling in the server. The goal of thermal management is to find cost-effective ways to maintain the junction temperature of the CPU lower than specifications and ensure the continuous operation of the server. Wei [1] has proved a 40 kW server can be cooled by forced air cooling.

However, it requires highly integrated design and a huge amount of air flow that the 54 fans inside PRIMEPOWER 2500 can generate. In the near future, it would be very difficult for a forced air-cooling method to cool cabinets with more than 60 kW power. It would require bigger fan trays to deliver huge amounts of air flow and large size heat sinks to transfer heat from the CPUs to air, which makes it impossible to design a reliable, compact and cost-effective cooling system for the server.

We have to find alternative ways to deal with this problem, Other cooling methods, such as air impinging jets, liquid cooling and refrigeration cooling systems, have the potential to dissipate more heat. But it will require intuitive packaging to integrate them into the server system.

References:

  1. Wen, J., Thermal Management of Fujitsu’s High-performance Servers, source: http://www.fujitsu.com/downloads/MAG/vol43-1/paper14.pdf.
  2. Koide, M.; Fukuzono, K.; Yoshimura, H.; Sato, T.; Abe, K.; Fujisaki, H.; High-Performance Flip-Chip BGA Technology Based on Thin-Core and Coreless Package Substrates, Proceedings of 56th ECTC, San Diego, CA, USA, 2006, pp.1869-1873.
  3. Xu, G; Follmer, L.; Thermal Solution Development for High-end System, Proceedings of 21st IEEE SEMI-THERM Symposium, San Jose, CA, USA, 2005, pp. 109-115.

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.

Immersion Liquid Cooling of Servers in Data Centers

A data center is a large infrastructure used to house large quantities of electronic equipment, such as computer servers, telecommunications equipment, and data storage systems, etc. The data center requires non-interrupted power, communication and internet access to all equipment inside, it also has dedicated environment control system which provides appropriate working conditions for the electrical devices hosted inside.

Immersion Cooling

Traditional data centers use cold air generated by a room air conditioner system (CRAC) to cool the servers installed on the racks. Cooling the electrical devices by cold air generated by an air conditioner is an easy method to implement. However, it is not a very efficient method in terms of power consumption.

The inefficiency of the method can be contributed to several causes: generating and delivering cold air from a chiller to servers is a multiple heat transfer process, such as the mixing of warm and cool air in the room, which reduces the efficiency and power consumption of cooling hardware such as chillers, computer room air conditioners (CRACs), fans, blowers and pumps.

Data center designers and operators have invented many ways to improve the data center’s thermal efficiency, such as optimizing the rack layout and air conditioner location, separating cold aisles and hot aisles, optimizing the configuration of pipes and cables in under-floor plenum, introducing liquid cooling to high-power severs.

While the above methods can improve the data center heat load management, they cannot dramatically reduce the Power Usage Effectiveness (PUE), which is a measure of how efficiently a datacenter uses its power and is defined as the ratio of total datacenter power consumption to the IT equipment power consumption.

An ideal PUE is 1,0. A better way, proposed and used by some new data centers, is directly bringing the outside cold air to the servers. This method can eliminate the computer room air conditioners (CRACs). To achieve this, the data center has to be located in a specific area where cold air can be provided for all four seasons and the servers have to have higher operating environmental temperature.

Another dramatic solution proposed and used by some companies is liquid immersion cooling for entire servers. When compared with traditional liquid cooling techniques, the liquid immersion cooling uses dielectric fluid as a working agent and open bath design. This eliminates the need for hermetic connectors, pressure vessels, seals and clamshells. There are several different liquid immersion cooling methods.

This article will review the active single-phase immersion cooling technology proposed by Green Revolution Cooling (GRC) [1] and a passive two-phase immersion cooling technology proposed by the 3M Company [2].

Green Revolution Cooling has designed a liquid-filled rack to accommodate the traditional servers and developed dielectric mineral oil as the coolant. Figure 1 shows the liquid cooling racks with chiller and an inside view of a CarnotJet cooling rack from GRC. The racks are filled with 250 gallons of dielectric fluid, called GreenDEF™, which is a non-toxic, clear mineral oil with light viscosity.

Figure 1. Server racks and chiller (left) and inside view of the server rack. [1]

The servers are installed vertically into slots inside the rack and fully submerged in the liquid coolant. Pumps are used to circulate the cold coolant from the chiller to the rack. The coolant returns to the chiller, after removing heat from the servers. Because of its high heat capacity and thermal conductivity, the GreenDEF™ can cool the servers more efficiently than air.

The server racks are semi-open to the environment and the coolant level is constantly monitored by the system. Figure 2 shows a server motherboard is being submerged in the coolant liquid inside a server rack from GRC.

Figure 2. A Server Motherboard Being Immersed in Liquid Coolant in A Server Rack. [1]

Intel has conducted a year-long test with immersion cooling equipment from Green Revolution Cooling in New Mexico [3]. They have found that the technology is highly efficient and safe for servers. In their tests, Intel tested two racks of identical servers – one using traditional air cooling and the other immersed in a Green Revolution enclosure. Over the course of a year, the submerged servers had a partial Power Usage Effectiveness (PUE) of 1.02 to 1.03, equaling some of the lowest efficiency ratings reported using that metric.

The 3M Company is also actively engaged in immersion cooling technology and has developed a passive two-phase immersion cooling system for servers. Figure 3 illustrates the concept of the immersion cooling system developed by 3M. In a specially designed server rack, servers are inserted vertically in the rack. The servers are immersed in 3M’s Novec engineered fluid, a non-conductive chemical with a low boiling point.

The elevated temperature of electronic components on the sever boards will cause the Novec engineered fluid to boil. The evaporation of the fluid will remove a large amount of heat from the heated components with small temperature difference. The evaporated fluid travels to the upper portion of the server rack, where it condenses to liquid on the surface of the heat exchanger cooled by the cold water. The condensed liquid flows back to the rack bath, driven by the force of gravity. In 3M’s server rack, the liquid bath is also semi-open to the outside environment.

Because the cooling method is passive, there is no pump needed in the system.

Figure 3. Passive Two-phase Immersion Cooling System from 3M. [2]

By utilizing the large latent heat of Novec engineered fluid during evaporation and condensation, the coolant can remove heat from servers and dissipate it to water heat exchanger with small a temperature gradient. To enhance the boiling on the component surfaces, 3M invented special coating for electronic chips inside the liquid bath. The boiling enhancement coating (BEC) is a 100 mm thick porous metallic material.

The application of the BEC is illustrated in Figure 4. The coating is directly applied to the integrated heat spreader (IHS) of the chip. Tuma [2] claimed that the coating can produce boiling heat transfer coefficients in excess of 100,000 W/m2-K, at heat fluxes exceeding 300,000 W/m2.

Figure 4. Application of boiling enhanced coating (BEC). [2]

In his paper, Tuma [2] discussed the economic and environmental merits of the passive two-phase immersion cooling technology for cooling data center equipment. He concluded that liquid immersion cooling can dramatically decrease the power consumption for cooling relative to traditional air-cooling methods. It can also simplify facility construction by reducing floor space requirements, eliminating the need for air cooling infrastructure such as plenum, air economizers, elevated ceilings etc.

Green Revolution Cooling and 3M have demonstrated the feasibility and applicability of using immersion cooling technology to cool the servers in data centers. The main advantages of immersion liquid cooling are saving overall cooling energy and maintaining the component temperature low and uniform. However, both immersion liquid cooling technologies require specially designed sever racks. Specially formulated coolants are needed for both cooling technologies, too, and they are not cheap. For the traditional air-cooled data center, the air is free, abundant and easy to deliver.

In both immersion cooling technologies, the servers have to be vertically installed inside the server rack, which will reduce the date center footprint usage efficiency. Because the liquid baths used in immersion cooling are open to the environment, coolant is gradually and inevitably lost to the ambient during long term service.

The environmental impact of the discharge of a large amount of coolant by data centers has to be evaluated, too. The effect of the coolant on the connectors and materials used on the PCB is not also very clear.

Immersion liquid cooling is a very promising technology for cooling high-power servers. But, there are still obstacles that need to be overcome before their large scale application is assured.   

References

  1. http://www.grcooling.com
  2. Tuma, E. P., “The Merits of Open Bath Immersion Cooling of Datacom Equipment,” 26th IEEE SEMI-THERM Symposium, Santa Clara, California, USA  2010.
  3. http://www.datacenterknowledge.com

For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit https://www.qats.com/consulting or contact ATS at 781.769.2800 or ats-hq@qats.com.