{"id":9967,"date":"2019-04-03T16:41:15","date_gmt":"2019-04-03T20:41:15","guid":{"rendered":"http:\/\/www.qats.com\/cms\/?p=9967"},"modified":"2025-09-26T15:31:28","modified_gmt":"2025-09-26T19:31:28","slug":"meeting-the-thermal-management-requirements-of-high-performance-servers","status":"publish","type":"post","link":"https:\/\/www.qats.com\/cms\/2019\/04\/03\/meeting-the-thermal-management-requirements-of-high-performance-servers\/","title":{"rendered":"Meeting the thermal management requirements of high-performance servers"},"content":{"rendered":"\n

High-performance servers are devices specially designed to handle large computational loads, a huge amount of communication signals, fast data processing, etc. Due to their task-oriented nature, high-performance servers must have high reliability, interchangeability, compact size and good serviceability. <\/p>\n\n\n

\n
\"\"
Photo by Taylor Vick<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n

To achieve high computational speed, high-performance servers generally have dozens of CPUs and memory models. They also have dedicated data process modules and control units to ensure seamless communication between CPUs and parallel data processing ability. To reach higher speeds, the power dissipation of high\u2013performance CPUs has been increasing continuously in the past decade for its use in high-performance servers.<\/p>\n\n\n\n

Cooling\ndozens of kW servers brings a unique challenge for thermal engineers. To deal\nwith the ever-growing high heat flux issue in high-performance servers, it will\nneed the cooperation of electrical, mechanical and system engineers to solve\nthe problem. The job to remove the high heat flux from CPUs to ambient requires\nchip level, board level and cabinet level solutions.<\/p>\n\n\n\n

Wei [1] described Fujitsu\u2019s thermal management advancements in their high-end UNIX server PRIMEPOWER 2500. The server cabinet is shown in Figure 1. Its dimension is 180cm \u00d7 107cm \u00d7 179cm (H\u00d7W\u00d7D) and has a maximum power dissipation of 40 kW. The system configuration of PRIMEPOWER 2500 is shown in Figures 2 and 3. It has 16 system boards and 2 input\/output (I\/O) boards installed vertically on two back-panel boards. The two back-panel boards are interconnected by six (6) crossbars installed horizontally.<\/p>\n\n\n

\n
\"\"
Figure 1. PRIMEPOWER 2500 Cabinet [1]<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Figure 2. PRIMEPOWER 2500 System Configuration [1]<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Figure 3. PRIMEPOWER 2500 System Board Unit [1]<\/figcaption><\/figure>\n<\/div>\n\n\n

To\ncool the electrical components inside PRIMEPOWER 2500, 48 200-mm-diameter fans\nare installed between the system board unit and the power supply unit. They\nprovide forced air cooling for system boards and power supplies. In addition,\nsix 140-mm-diameter fans are installed on one side of crossbar to cool the\ncrossbar boards with a horizontal flow. The flow direction is shown in Figure\n3. Each system board is 58 cm wide and 47 cm long. <\/p>\n\n\n\n

There are eight CPU processors, 32 Dual In-Line Memory Modules, 15 system controller processors, and associated DC-DC converters on each system board. The combined power dissipation per system board is 1.6 kW at most.<\/p>\n\n\n

\n
\"\"
Figure 4. PRIMEPOWER 2500 System Board [1]<\/figcaption><\/figure>\n<\/div>\n\n\n

To\ncool the electrical components inside PRIMEPOWER 2500, 48 200-mm-diameter fans\nare installed between the system board unit and the power supply unit. They\nprovide forced air cooling for system boards and power supplies. In addition,\nsix 140-mm-diameter fans are installed on one side of crossbar to cool the\ncrossbar boards with a horizontal flow. The flow direction is shown in Figure\n3. Each system board is 58 cm wide and 47 cm long. <\/p>\n\n\n\n

There\nare eight CPU processors, 32 Dual In-Line Memory Modules, 15 system controller\nprocessors, and associated DC-DC converters on each system board. The combined\npower dissipation per system board is 1.6 kW at most.<\/p>\n\n\n\n

Forced\nair-cooling technology is commonly used in computers, communication cabinets,\nand embedded systems, due to its simplicity, low cost and easy implementation.\nFor high-performance servers, the increasing power density and constraints of air-cooling\ncapability and air delivery capacity have pushed forced air cooling to its\nperformance limit. <\/p>\n\n\n\n

For\nhigh power systems like PRIMEPOWER 2500, it needs a combination of good CPU\ndesign, optimized board layout, advanced thermal interface material (TIM),\nhigh-performance heat sinks, and strong fans to achieve desired cooling.<\/p>\n\n\n\n

The\ngeneral approach to cool the multi-board system is first to identify the\nhottest power component with the lowest temperature margin. For the\nhigh-performance server, it is the CPUs. For multiple CPUs on a system board,\ngenerally, the CPU located on downstream of a board or other CPUs has the\nhighest temperature. <\/p>\n\n\n\n

So, the thermal resistance requirement for this CPU is:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Where Tj,max<\/sub> is the allowed maximum junction\ntemperature, Ta<\/sub> is the ambient temperature, \u2206Ta<\/sub> is the air temperature rise due to\npreheating before the CPU, and qmax<\/sub> is the maximum CPU power.<\/p>\n\n\n\n

The junction-to-air thermal resistance of the CPU is:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Where Rjc<\/sub> is the CPU junction-to-case thermal\nresistance, RTIM<\/sub> is the thermal resistance of thermal interface\nmaterials, and Rhs<\/sub> is the heat sink thermal resistance. To reduce\nthe CPU junction temperature, it is critical to find intuitive ways to minimize\nRjc<\/sub>, RTIM<\/sub>, and Rhs<\/sub>, because any reduction in\nthermal resistance is important in junction temperature reduction.<\/p>\n\n\n\n

The CPU package and heat sink module of PRIMEPOWER 2500 are shown in Figure 5. The CPU package has an integrated heat spreader (IHS) attached to the CPU chip. A high-performance TIM is used to bond the CPU chip and IHS together, see Figure 6. The heat sink module is mounted on the IHS with another TIM in between.<\/p>\n\n\n

\n
\"\"
Figure 5. PRIMEPOWER 2500 CPU Package and Heat Sink Module [1]
<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Figure 6. CPU Package [1]<\/figcaption><\/figure>\n<\/div>\n\n\n

The\nTIM used in between the CPU chip and the IHS are crucial to the CPU\u2019s\noperation. It has two key functions: to conduct heat from the chip to the IHS\nand to reduce the CPU chip stress caused by the mismatch of the coefficient of\nthermal expansion (CTE) between the CPU chip and IHS. Fujitsu developed a TIM\nmade of In-Ag composite solder for the above application. The In-Ag composite\nhas a low melting point and a high thermal conductivity. It is relatively soft,\nwhich is good for absorbing thermal stress between the chip and the IHS.<\/p>\n\n\n\n

Wei [2] also investigated the impact of thermal conductivity on heat spread performance. He found a diamond composite IHS (k=600 W\/(mK)) would result in a lower temperature gradient across the chip and low temperature hot spots, compared with aluminum nitride (k=200 W\/(mK)) and copper (k=400 W\/(mK)). The simulation results are shown in Figure 7.<\/p>\n\n\n

\n
\"\"
Figure 7. Heat Spreader Material Comparison [2]<\/figcaption><\/figure>\n<\/div>\n\n\n

In\nhigh-performance servers like the PRIMEPOWER 2500, the thermal performance\ngains by optimizing the TIM and the IHS are small, because they compose only a\nsmall portion of the total thermal resistance. Heat sinks dissipate heat from\nthe CPU to air and have an important role in the thermal management of the\nserver. In a server application, the heat sink needs to meet not only the\nmechanical and thermal requirements, but also the weight and volume restraints.\nHence, heat pipes, vapor chambers, and composite materials are widely used in\nplace of high-performance heat sinks.<\/p>\n\n\n\n

Koide\net al [1] compared the thermal performance and weight of different heat sinks\nfor server application. The results are shown Figure 8. They used the\nCu-base\/AL-fin heat sink as benchmark. Compared with the Cu-base\/AL-fin heat\nsink, the Cu-base\/Cu-fin heat sink is 50% heavier and gains only 8%\nperformance. <\/p>\n\n\n\n

If the heat pipe is used in base, the heat sink weight can be reduced by 15% and the thermal performance increases by 10%. If the vapor chamber is embedded in the heat sink base, it reduces the heat sink weight by 20% and increases the heat sink performance by 20%.<\/p>\n\n\n

\n
\"\"
Figure 8. Thermal Performance and Weight Comparison of Different Heat Sinks [1]<\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Figure 9. (a) USIII Heat Sink for Sun Fire 15K Server, (b) USIV Heat Sink for Sun Fire 25K [3]<\/figcaption><\/figure>\n<\/div>\n\n\n

Sun Microsystems\u2019 high-performance Sun Fire 15K Server uses USIII heat sink to cool its 72 UltraSparc III (USIII) processors. In Sun Fire 25K Server, the CPUs are upgraded to UltraSparc IV (USIV), which has a maximum power of 108 W. To cool the USIV processor, Xu and Follmer [3] designed a new USIV heat sink with copper base\/copper fin, see Figure 9. The old USIII heat sink has 17 forged aluminum fins, the USIV heat sink has 33 copper fins. Both heat sinks have the same base dimensions and height.<\/p>\n\n\n

\n
\"\"
Figure 10. Thermal Resistance Comparison between USIII Heat Sink and USIV Heat Sink [3]<\/figcaption><\/figure>\n<\/div>\n\n\n

Figure\n10 shows the thermal resistance comparison between the USIII heat sink and the USIV\nheat sink. The thermal resistance of the USIV heat sink is almost 0.1\u00b0C\/W lower\nthan that of the USIII heat sink at medium and high flow rates, which is a huge\ngain in thermal performance. The thermal performance improvement of the USIV heat\nsink is not without penalty. <\/p>\n\n\n

\n
\"\"
Figure 11. Pressure Drop Comparison between USIII Heat Sink and USIV Heat Sink [3]<\/figcaption><\/figure>\n<\/div>\n\n\n

Figure\n11 shows the pressure drop comparison between the USIII heat sink and the USIV heat\nsink. For the same air flow rate, the pressure drop of the USIV heat sink is\nhigher than that of the USIII heat sink. That means the Sun Fire 25K Server\nneeds stronger fans and better flow arrangements to ensure the USIV heat sinks\nhave adequate cooling flow.<\/p>\n\n\n\n

The\ndesign of the cooling method in high-performance servers follows the same\nmethodology used in the design cooling solution of other electronic devices,\nbut at an elevated scale. The main focus is to identify the hottest components,\nwhich in most cases is CPUs. Due to extreme high power of CPUs, memory modules,\ncheat spreader, TIM, and heat sinks to achieve desired cooling in the server. The\ngoal of thermal management is to find cost-effective ways to maintain the\njunction temperature of the CPU lower than specifications and ensure the\ncontinuous operation of the server. Wei [1] has proved a 40 kW server can be\ncooled by forced air cooling. <\/p>\n\n\n\n

However,\nit requires highly integrated design and a huge amount of air flow that the 54\nfans inside PRIMEPOWER 2500 can generate. In the near future, it would be very\ndifficult for a forced air-cooling method to cool cabinets with more than 60 kW\npower. It would require bigger fan trays to deliver huge amounts of air flow\nand large size heat sinks to transfer heat from the CPUs to air, which makes it\nimpossible to design a reliable, compact and cost-effective cooling system for\nthe server. <\/p>\n\n\n\n

We\nhave to find alternative ways to deal with this problem, Other cooling methods,\nsuch as air impinging jets, liquid cooling and refrigeration cooling systems,\nhave the potential to dissipate more heat. But it will require intuitive\npackaging to integrate them into the server system.<\/p>\n\n\n\n

References:<\/strong><\/p>\n\n\n\n

    \n
  1. Wen, J., Thermal Management of Fujitsu\u2019s High-performance
    Servers, source: http:\/\/www.fujitsu.com\/downloads\/MAG\/vol43-1\/paper14.pdf<\/a>.
    <\/li>\n\n\n\n
  2. Koide, M.; Fukuzono, K.; Yoshimura, H.; Sato, T.; Abe, K.;
    Fujisaki, H.;     High-Performance
    Flip-Chip BGA Technology Based on Thin-Core and Coreless Package Substrates<\/a>,
    Proceedings of 56th ECTC, San Diego, CA, USA, 2006, pp.1869-1873.<\/li>\n\n\n\n
  3. Xu, G; Follmer, L.; Thermal Solution
    Development for High-end System<\/a>, Proceedings of 21st IEEE SEMI-THERM
    Symposium, San Jose, CA, USA, 2005, pp. 109-115.<\/li>\n<\/ol>\n\n\n\n
    \n\n\n\n

    For more information about Advanced Thermal Solutions, Inc. (ATS) thermal management consulting and design services, visit <\/em>https:\/\/www.qats.com\/consulting<\/em><\/a> or contact ATS at 781.769.2800 or <\/em>ats-hq@qats.com<\/em><\/a>.\ufeff<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

    High-performance servers are devices specially designed to handle large computational loads, a huge amount of communication signals, fast data processing, etc. Due to their task-oriented nature, high-performance servers must have high reliability, interchangeability, compact size and good serviceability. To achieve … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1001004,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,687],"tags":[914,101,922],"class_list":["post-9967","post","type-post","status-publish","format-standard","hentry","category-datacenter","category-servers","tag-data-centers","tag-heat-sinks-2","tag-servers"],"_links":{"self":[{"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/posts\/9967","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/users\/1001004"}],"replies":[{"embeddable":true,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/comments?post=9967"}],"version-history":[{"count":6,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/posts\/9967\/revisions"}],"predecessor-version":[{"id":12481,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/posts\/9967\/revisions\/12481"}],"wp:attachment":[{"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/media?parent=9967"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/categories?post=9967"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.qats.com\/cms\/wp-json\/wp\/v2\/tags?post=9967"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}